The Language of Extended Reality Technologies and Experiences

Katerina Girginova

Matters of language in research

The goal behind this section of the report is to help us better communicate about extended reality^[1] (XR) media and our experiences with it. This is done by examining how a range of key academic articles published in 2024 conceptualize XR. Of particular interest is the context in which XR appears and new terminology or explicit questions of communication that emerge. By highlighting interesting concepts beyond popular terms like ‘immersion’ and ‘presence’, which continue to define much of XR research^[2], we are able to more fully explore the breadth and depth of XR experiences and build a useful toolbox of vocabulary while doing so.

Methods of Analysis

This section draws from 306 peer-reviewed articles published in 2024 across three key journals dedicated to XR research^[3] plus the addition of a relevant article found beyond this scope. The three key journals examined here are the same as those examined throughout the other sections of the report and they were chosen because they are peer-reviewed publications, specifically dedicated to XR. The articles were read and inductively coded by the author. In some cases, the focus of the articles was communicative; that is, the author’s intent was specifically to examine questions of meaning-making through language or, to introduce new terms to help us better communicate about XR. In other cases, the focus of the articles was not overtly related to communication matters. Still, in these cases, the language used to talk about XR is telling of the state and exploration taking place in XR research.

Key Findings

While the seemingly perennial topics of using VR for education and measuring and addressing motion sickness and presence remain pervasive across many of the articles surveyed, the 2024 cohort of publications shows a much wider array of XR subject matter and contexts than previous years. This ranges from multi-sensory explorations of XR media and connections to psychedelics, to questions of equity in virtual production and examples of tokens that can guide us back from virtual experiences to reality.

In terms of specific language used to describe XR, there remains a limited pool of terms used, and authors cited, much like in previous years of this report. Nonetheless, given the wider pool of topics and contexts explored in 2024, there emerge two interrelated themes (even if not predominant amongst the articles) and some useful new vocabulary to add to our XR research toolbox. These themes 1) push at the peripheries of current XR experiences through a renewed multi-sensory focus and 2) offer a deeper dive into virtual production, a specific context of XR work. As for the new vocabulary that emerges, terms like ‘tokens of reality’ and ‘numadelic aesthetic’ (described in more detail in the annotated bibliography below), can offer new ways to think about XR and inspire avenues for research. Furthermore, this vocabulary may also better qualify how immersion and presence takes place – topics, that despite their popularity, continue to spark debate.

The two themes that emerge from 2024’s cohort of articles are described below and then illustrated in more depth through the annotated bibliographies.

1. Theme 1: A renewed multi-sensory focus. A number of articles argue that our conscious realities are mediated through our senses (Glowacki, 2024; Hancock, 2024; Lopes et al., 2024; Sawahata, Harasawa & Komine, 2024; Sun et al., 2024) and so, they push for an extension of current XR experiences into multi-sensory directions that depart from present-day aesthetics. In turn, some of these articles are also able to ask bigger, bolder and ethically inquisitive questions like what happens when we reach the seemingly end goal of making virtual experiences indistinguishable from reality? While these questions themselves may not be new, their answers offer perspectives that are informed by new developments.
2. Theme 2: Practical reflections on virtual production practices. This theme reflects the growing practice of virtual production in the media and entertainment sectors and was partially bolstered by a special issue on Immersive Visualization in the journal Presence: Virtual and Augmented Reality. The articles grouped under this theme presented a reflection on the history of XR by explicitly detailing how virtual production and visualization works, what past and future developments entail, and what the un/successful practices and promises underpinning the practice were (See for example: Swords & Willment, 2024; Wischgoll, 2024). The names and acronyms given to specific technologies (CAVE, A.R.T, GeoWall, AccessBot, etc.,) are telling, too, and echo the linguistic evolution of their designer’s visions throughout time. In addition, the need to build a vocabulary that reflects the hybrid yet unique practices underpinning virtual production emerges, which further supports recent industry reports calling for the same development (Bennett et al., 2024; Bennett & Murphy, 2020).

Annotated Bibliography: A selection of articles showcasing the themes

Hancock, P. A. (2024). Tokens of Reality: On the Prospective Nature of Virtual Consciousness. PRESENCE: Virtual and Augmented Reality, 33, 469-480.

This article introduces the concept of tokens of reality to ask what will be the talisman that guides our return to that reality when our technology becomes so advanced that it is impossible to distinguish its productions from reality? Furthermore, it asks the important question: what will we have achieved when we overcome the technical ‘barriers’ to get to this seemingly desired state?

To set up the search for individual tokens of reality, the author argues that our conscious realities are idiosyncratically mediated through our senses and introduces the term perceptual display augmentation to describe the XR technologies we create that further mediate our perception. Nevertheless, Hancock notes that the pace of development for perceptual display augmentation does not move in sync with our own evolutionary development to adapt to these technologies, hence, the need for tokens of reality. Ultimately Hancock proposes the development of neuroactive patterns of cognition as the appropriate tokens of reality – a reverse Turing Test – arguing that material tokens will be insufficient.

By critically introducing the term tokens of reality into the language of XR experiences, this article averts the predominant preoccupation with fixing glitches and improving technology to focus on a potential future where we live amongst infinite and indistinguishable virtualities. In addition, the author raises important questions of trust, such as those experienced around the current trend of deepfakes. In doing so, the author encourages readers to think of disorientation beyond physical reactions to XR experiences, like motion sickness, and to center on re-orientation techniques – a useful exercise even in the present.

Swords, J., & Willment, N. (2024). The emergence of virtual production–a research agenda. Convergence, 30(5), 1557-1574.

This article examines the rapidly emerging practice of virtual production in the film and television industries and the resulting new practices, possibilities and problems it creates. The author offers three ways in which virtual production is being adopted: 1) through live action with a green or blue screen, 2) through LED screens for virtual sets, and 3) through filming within virtual worlds. The primary benefits of virtual filmmaking are efficiencies in production and cost cutting. Lower environmental costs and greater work-life flexibility for those involved are also often mentioned in the industry, however, as Swords and Willment note, those two benefits are more questionable. In addition, the high upfront costs and learning curves which presently limit access to virtual production to high-end studios, make it an exclusive practice despite the inclusive rhetoric.

The authors call virtual production practices in film and television an “epochal shift”: “An ecosystem of technologies and workflows which sit at the intersection of established film and TV visual effects and immersive media and technologies,” (p. 1558). The following quote from a virtual production supervisor exemplifies the unique synergy between film, tv, visual effects and newer immersive media: “at its simplest form, we’re building a game called “Filmmaking” (Rogers, 2020 – quoted in Swords & Wilmment, p.g. 1560). While this article does not introduce new terms per se, it charts new territories of extended reality practice and notes that due to the fast pace at which virtual production is developing, communication issues emerge: “the different experience and industry backgrounds of crew members brought together for virtual production can create vocabulary problems and the novelness of workflows, equipment, software and other technologies means new language is being created,” (p. 1568). In turn, creating a shared set of vocabulary terms that describes virtual production practices is important for a range of factors from training – as the article and numerous reports note the demand for qualified workers in virtual production is far outstripping the supply – to evaluation of work and IP matters.

Glowacki, D. R. (2024). VR models of death and psychedelics: an aesthetic paradigm for design beyond day-to-day phenomenology. Frontiers in Virtual Reality, 4, 1286950.

This article introduces the term numadelic aesthetic to describe visualizations in VR that present bodies as light energy as opposed to physical form. Based on the author’s own near-death experience, the numadelic aesthetic seeks to create visual experiences that surpass the barriers of rational language to elicit therapeutic feelings that reduce anxiety around death, akin to those experienced through psychedelic drugs or near-death experiences. Glowacki also theorizes the numadelic aesthetic using knowledge from psychology, phenomenology and neuroscience to understand why it achieves the results it does.

More broadly, Glowacki argues that extended reality aesthetics are very limited to mimicking real-world scenarios that rely on pre-existing conceptions we have of how the world around us should look and feel, but there is benefit in breaking beyond these narrow confines. The word ‘numadelic’ derives from the Greek words ‘pneuma’ (breath or spirit) and ‘delein’ (to reveal) (p. 2). To theorize the numadelic, Glowacki creates a quadrant which charts along one axis structural specificity and along the other symbolic rigidity. On one end of the structural specificity axis lies infinite structural possibility of a given image/aesthetic that can accommodate ample interpretation, and on the other end lies a fully collapsed or restricted image/aesthetic that leaves little room for interpretation. Along the symbolic representation axis lies high symbolic rigidity, on one end (again limiting room for interpretation), and on the other end, low symbolic rigidity, opening space for various meaning-making. According to Glowacki, the numadelic aesthetic is an un-collapsed approach to meaning-making, which is characterized by low structural specificity and low symbolic rigidity; concepts, which can be useful in interpreting extended reality aesthetics beyond this piece.

References

Bennett, J., Bryant, K., Dalton, P., Heath, C., Glushkova, K., Levstek, M., Lycett, M., Njoku, E., Saunders, W., Smyth, S., Wills, H., Whittaker, L., & Woods, A. (2024). StoryFutures Creative Industries Clusters Final Report. StoryFutures.com. https://www.storyfutures.com/uploads/docs/StoryFutures_Cluster_Final_Report_2024_Final-compressed.pdf
Bennett, J., & Murphy, A. (2020). Skills for immersive experience creation. StoryFutures Academy, London. https://www.storyfutures.com/uploads/images/SFICC-Report-2019-20.2.20.pdf
Lopes, M. K. S., De Jesus Jr, B. J., Rosanne, O. M., Pardini, S., Appel, L., Smith, C., & Falk, T. H. (2024). Stop to smell the virtual roses: a mixed-methods pilot study on the impact of multisensory virtual reality nature experiences on feelings of relaxation. Frontiers in Virtual Reality, 5, 1451704.
Rogers, S. (2020, January 29). Virtual Production and the future of filmmaking – an interview with Ben Grossmann Magnopus. Forbes.com.
Sawahata, Y., Harasawa, M., & Komine, K. (2024). Synergy and medial effects of multimodal cueing with auditory and electrostatic force stimuli on visual field guidance in 360° VR. Frontiers in Virtual Reality, 5, 1379351.
Sun, W., Banakou, D., Świdrak, J., Valori, I., Slater, M., & Fairhurst, M. T. (2024). Multisensory experiences of affective touch in virtual reality enhance engagement, body ownership, pleasantness, and arousal modulation. Virtual Reality, 28(4), 1-16.
Wischgoll, T. (2024). Center for cyber-physical systems: Immersive visualization and simulation environment. PRESENCE: Virtual and Augmented Reality, 33, 13-29.

1. Extended reality, or XR, is an umbrella term which refers to mixed, augmented, and virtual reality technologies amongst others. In this case, however, it almost exclusively focuses on augmented and virtual reality, which were the key topics of the articles covered. ↑

2. See section on Theories in XR for a detailed discussion about the application of these terms. ↑

3. The three dedicated journals to XR research include Virtual Reality (178 articles across 4 issues), Frontiers in Virtual Reality (105 articles across 2 issues), and Presence: Virtual and Augmented Reality (23 articles across 2 issues, one of which was a thematic issue). The additional article comes from the journal Convergence and was chosen from a separate search due to its relevance to the topic. ↑