The Language of Extended Reality Technologies and Experiences

Katerina Girginova

As research has shown, the ways in which we think and communicate about something, and the specific language we use to do so, directly shapes our experiences of it (Berger & Luckmann, 1991^[1]; Searle, 1999^[2]). In turn, the goal behind this section of the report is twofold: first, to better understand how we communicate about extended reality^[3] (XR) media and experiences by examining the language used to describe them and second, to pursue innovations in the ways we conceptualize our interactions with XR media through the introduction of novel terms and concepts. This section draws from 266 peer-reviewed articles published in 2023 across four key journals dedicated to XR research^[4] plus the addition of five relevant articles found beyond this scope. The four key journals, which all of the scholarly sections throughout this report analyze, were selected because they are dedicated to publishing XR work; the additional articles were based on the author’s search.

One key finding is that in 2023 there is a very limited set of terms used and authors cited to describe XR media and experiences. Even articles that are focused on communicative aspects of XR, or are more qualitative or exploratory in nature, still adhere to the centripetal pull of psychological concepts like presence (Slater & Wilbur, 1997^[5]), embodiment, and immersion (Biocca & Delaney^[6], 1995; Biocca, 1997^[7]) to explain XR media or to study specific phenomena using XR. This is similar to findings from the 2022 report on the language of XR that found research to be anchored around the same key terms without necessarily doing the work of interpretation or explanation. One might think this is to be expected, particularly given the objectivist epistemology that largely underpins the key journals from which the corpus of articles is drawn and yet, there is reason for change.

While a limited pool of descriptive terms is not inherently problematic, the ways in which we communicate about XR experiences should reflect the evolutions in our creation and consumption practices around XR technologies. Furthermore, as previous research has aptly pointed out, there are varied intellectual traditions and important differentiating markers even within seemingly taken-for-granted key terms like “presence” (Felton & Jackson, 2022^[8]; Murphy & Skarbez, 2020^[9]), which current research is rarely specific about. In short, there is a need if not for more terminology, then certainly for more clarity on what exactly we mean when we communicate about XR.

While not a dominating trend throughout the corpus of articles, three interrelated themes also emerge: 1) a focus on narrative, 2) sensory/spatial considerations, and 3) augmented reality. Specifically, the exploration of narrative extends to how the human senses and our immediate, physical surroundings become a part of the overall virtual reality (VR) experience, the co-creation of meaning between audience and performer and user and virtual artifact, the veracity of XR media narratives, and descriptions of the metaverse. Augmented reality (AR) was also the key topic of several articles, which offer readers empirical concepts to better depict our engagement with AR. The annotated bibliography below showcases each of these themes in more detail via a selection of articles.

Innovations in the language of XR include the introduction and systematic examination of “onboarding” and “offboarding” practices into XR media, the metaphor of the museum to structure how we think about the encoding and decoding of meaning in VR media, and the provocative suggestion to rename VR to “sensory immersion” or SI. Last, it’s worth noting that the concept of Artificial Intelligence (AI) did feature in some of the articles but was not a prominent theme and when it was mentioned, it was often in hypothetical reference to what it could mean for future XR content creation. Similarly, spatial computing did not enter the literature in a meaningful way, highlighting the lag between academia and industry.

Annotated Bibliography: A selection of articles showcasing the themes of narrative, spatiality, the human senses, and augmented reality

Jang, S. Y., Park, J., Engberg, M., MacIntyre, B., & Bolter, J. D. (2023). RealityMedia: immersive technology and narrative space. Frontiers in Virtual Reality, 4. https://doi.org/10.3389/frvir.2023.1155700

This paper examines the meaning-making processes behind the consumption of non-fiction VR narratives. It designs a VR experiment, in which 20 people are shown a museum-like experience called RealityMedia, with various galleries showcasing the history of media. The authors argue that VR “is a new writing space in the long tradition of inscription” (p. 1), in which the consumption of narratives is akin to the sensemaking that takes place in a museum. Specifically, the authors posit that narrative sense-making occurs on three levels: the architecture of the space, the collection, and the individual artifacts.

This article presents an interesting companion to the book Reality Media^[10] (2021), which explores the co-existence of older and newer media forms and the ways in which newer media like television remediate older media like radio. The authors highlight the added complexity of designing VR narratives, which now take place across visual, auditory, and spatial (and in some cases haptic) dimensions. They also draw attention to the tension of authorial control versus user agency in the level to which VR narratives need to be guided to appeal to different audiences. In turn, the metaphor of the museum and the three specific layers of meaning inscription and decoding emerge as tools for thinking about VR narratives.

Harley, D. (2023). Virtual narratives, physical bodies: Designing diegetic sensory experiences for virtual reality. Convergence, https://doi.org/10.1177/13548565231178915

This article uses a “research through design” approach (Gaver, 2012^[11]) to examine how sensory experiences can be incorporated into diegetic^[12] VR narratives, whereby the desired contribution of the paper is to use design “to produce questions rather than products” (p. x). As an illustration, the author discusses four specific projects that contextually insert various physical and sensory interactions into the VR experience, such as hand controllers that have been adapted to the shape and feel of a specific object, or sand beneath the user’s bare feet. The paper prompts researchers and designers to expand beyond the more traditional visual and auditory narrative form, by considering how design elements in one’s immediate surrounding, like the VR headset s/he is wearing or other, low-tech real objects, can be meaningfully incorporated into the overall XR experience. In turn, such a practical and conceptual expansion would also result in a reorientation of the producer’s “design space.”

Whittaker, L. (2023). Onboarding and offboarding in virtual reality: A user-centred framework for audience experience across genres and spaces. Convergence, https://doi.org/10.1177/13548565231187329

This paper, which is part of the expansive Story Futures project^[13], is the first systematic consideration of onboarding and offboarding into VR experiences. The author defines onboarding and offboarding as the processes of ushering audiences into an immersive experience and backing out again. While the author notes that these practices have been extensively examined in the settings of theater and live performance, they tend to be neglected in the context of VR—to the detriment of those using it. Furthermore, the concepts of immersion and presence, key features of VR media, also tend to gloss over the processes necessary to get into and out of those states. Thus, including onboarding and offboarding as vital concepts to the work of immersion and presence requires us to expand our understandings of these latter VR keywords.

The author presents an adapted framework from the StoryFutures Audience Toolkit for inviting the “VR audiences into a contract of participation” (p.13). The framework asks those designing VR experiences to consider their responsibility to the audience and their desired effect via suitable onboarding and offboarding practices based on a consideration of platform, place, time, genre, and user. Finally, the author advocates for two shifts: 1) one which focuses attention away from technology and toward the user, and 2) one which moves beyond the “moment of immersion to the process of incorporation” (p. 17). The latter concept builds upon arguments that people are frequently aware of their physical surroundings even when experiencing absorbing VR content, thus onboarding and offboarding becomes a question of how to optimally lead audiences into these states of constant perceptual negotiation rather than into a final destination.

Kirwan, E. (2022). Performer/audience experience, performer perception and audience immersion. Virtual Creativity, 12(1). https://doi.org/10.1386/vcr_00060_1

This article examines three VR-mediated dance performances and juxtaposes the experiences of the audience and performers. The author argues that VR “enhances the performer’s and audience’s immersion, yet by doing so, separates performer and audience even further” (p. 55), leading to a non-event that is more akin to watching a film than experiencing the co-creation of meaning during live performance. As such, the article offers reflections on the evolution of dance, as a story medium that “immerses audiences through its unfolding narrative” (p. 52).

Bay, M. (2023). Arendt in the Metaverse: Four properties of eXtended Reality that imperil factual truth and democracy. Convergence, 29(6), 1698-1712. https://doi.org/10.1177/13548565231199957

This article argues that extended reality (XR) technologies inhibit the social creation of what Arendt (1968^[14]) calls a common reality and factual truth because they are increasingly connected to large media conglomerates that favor profitable audience segmentation and content personalization. The author argues this personalization is particularly potent when presented via XR content, which has unprecedented powers to persuade. While the author acknowledges that the article makes an abstracted, and somewhat theoretical argument, the synergies between generative AI that can create photorealistic virtual reality content and the impact upon the audiences who consume it warrant attention.

The problem with high levels of content personalization in XR—similar to other media—is that there becomes “less basis for political interaction and deliberation—and hence, the co-construction of a common reality is inhibited” (p. 1705). However, unlike other media, hypertargeted personalization in XR can lead to some unique issues including false memory creation, not only on cognitive but also on motoric levels, and predictive processing problems. The latter is premised upon the fact that our present perception is based on our previous experiences, hence we may feel “XR-induced false memories even when… not using any XR technology at all” (p. 1707). The article concludes with a broader contextualization of XR within the debate around misinformation, false narratives and regulation. Provocatively, the author notes that AR-enhanced reality is typically not considered misinformation, but it is a space that needs close monitoring: “any regulator who is concerned about the democratic threat of mis-/disinformation mut recognize the challenge of individualized reality misperceptions such as those made possible by AR glasses” (p. 1708).

Boellstorff, T. (2024). Toward anthropologies of the metaverse. American Ethnologist, 51(1), 47-56. DOI: 10.1111/amet.13228

This article argues that anthropology, which has long studied socio-cultural practices in a range of non-mediated and mediated settings, can help us untangle the presently abundant misconceptions about the metaverse. Specifically, the author identifies four optional characteristics of the metaverse that are frequently and misleadingly described as necessary features: first, that VR is necessary to access the metaverse. Boellstorff gives the example of Second Life as a popular metaverse or virtual place that does not require VR. To underscore the proposed separation of the metaverse from VR, the author argues that “what makes the metaverse real is social immersion not sensory immersion” (p. 4). Further, Boellstorff suggests that since VR is primarily a sensory medium, it should more accurately be renamed “sensory immersion” while reserving the term virtual reality for broader consumption.

Second, the author dismantles interoperability between metaverses as a benefit; to the contrary, he argues that there is value in being able to separate one’s identity in different online as well as offline settings. Third, the need for a large scale metaverse is debunked as there is value in various-sized communities—including smaller fringe ones and lastly, crypto currency is highlighted as yet another optional feature. The author argues that the framing of these optional characteristics as fundamental to a metaverse primarily benefits the big corporations building it, and further, that anthropology has valuable tools to help us decolonize future visions from promotional rhetoric and technological “lock-in.”

This piece also tackles the notion that the metaverse “is already passé—supplanted by generative artificial intelligence like ChatGPT” (p. 1). In response, the author suggests that uninspired corporate visions have contributed to pushing the metaverse (and arguably VR has had a similar fate) into a hype cycle that inevitably involves becoming supplanted by the next big technology. Yet, the metaverse’s history dates back to the 19^th century telegraph and will continue to evolve as a place for virtual interaction “linked to physical world-place” (p. 7).

Alha, K., Leorke, D., Koskinen, E., & Paavilainen, J. (2023). Augmented play: An analysis of augmented reality features in location-based games. Convergence, 29(2), 342-361. https://doi.org/10.1177/13548565231156495

This article analyzes how the physical environment is incorporated within 11 augmented reality (AR) games. The authors uncover superimposition, blending, immersivity, and materiality as the key processes through which this takes place; however, they note that consistent with previous research, AR remains more of a marketing strategy or “gimmick” than a central part of the games. In brief, superimposition shows game content on top of the phone camera’s view without responding directly to the surrounding environment. Blending is dependent upon the phone’s camera to recognize suitable locations for game content to appear. Immersion occurs when the game shows the player content around them, making the player feel as if they are within the game, and materiality occurs when the game recognizes physical objects with the in-game camera and subsequently brings them into the game.

At present, AR’s social imaginary far supersedes its actual uptake, and broader audiences are limited due to existing social practices around gameplay and high levels of technological errors with AR. Yet, the authors use their findings to also question the present definition of AR: “if we instead defined AR as the broader connection of virtual information to physical space in real-time, every location-based game could potentially be considered as ‘augmented reality’” (p. 358). Furthermore, AR gaming is just one step in the broader social “shift towards ‘augmented space’” (p. 344).

1. Berger, P. L., & Luckmann, T. (1991). The social construction of reality: A treatise in the sociology of knowledge (No. 10). London, UK: Penguin UK. (Originally published 1966). ↑

2. Searle, J. R. (1995). The construction of social reality. New York, NY: Simon and Schuster. ↑

3. While extended reality of XR as an umbrella term refers to mixed, augmented, and virtual reality technologies the articles almost exclusively focused on the latter two. ↑

4. The four dedicated journals to XR research include Virtual Reality (153 articles), Frontiers in Virtual Reality (91 articles), Presence: Virtual and Augmented Reality (13 articles), and Virtual Creativity (13 articles). The five additional articles come from the journals Convergence and American Ethnologist. ↑

5. Slater, M., and Wilbur, S. (1997). A Framework for Immersive Virtual Environments (FIVE): Speculations on the Role of Presence in Virtual Environments. Presence: Teleoperators & Virtual Environments 6, 603–616. ↑

6. Biocca, F., & Delaney, B. (1995). Immersive virtual reality technology. Communication in the age of virtual reality, 15(32), 10-5555. ↑

7. Biocca, F. (1997). The cyborg's dilemma: Progressive embodiment in virtual environments. Journal of computer-mediated communication, 3(2), JCMC324. ↑

8. Felton, W. M., & Jackson, R. E. (2022). Presence: A review. International Journal of Human–Computer Interaction, 38(1), 1-18. ↑

9. Murphy, D., & Skarbez, R. (2020). What Do We Mean When We Say “Presence”?. PRESENCE: Virtual and Augmented Reality, 29, 171-190. ↑

10. Bolter, J. D., Engberg, M., & MacIntyre, B. (2021). Reality media: Augmented and virtual reality. MIT Press. ↑

11. Gaver, W. (2012, May). What should we expect from research through design?. In Proceedings of the SIGCHI conference on human factors in computing systems (pp. 937-946). ↑

12. Diegetic narrative often refers to the sound that occurs within the context of a story and that can be heard by its characters. In the context of this article, however, diegetic narrative refers to any sensory characteristics or interactions situated in the storyworld that the user directly experiences. ↑

13. Bennett, J., Dalton, P., Goriunova, O., Preece, C., Whittaker, L., Verhulst, I., & Woods, A. (2021). The story of immersive users. StoryFutures Audience Insight Report. ↑

14. Arendt, H., (1968). Between Past and Future: Eight Exercises in Political Thought. New York, NY: The Viking Press. ↑