The Language of XR

Katerina Girginova

This section examines the language researchers use to communicate about extended reality (XR) media^[1]. This was prompted, in part, by a perceived regurgitation of key terms used to describe XR media interactions. Perhaps, this is due to a limitation of available concepts, which may be self-imposed, to describe our complex interactions with this emergent media form. Interactions, which arguably demand more physicality and a higher level of sensorial engagement from the audience than any previous media form. In equal part, this section comes about via a desire to critically push back against some of the terminology being popularized by corporations. Encouragingly, this desire was echoed in some of the publications examined. In turn, a survey of the articles within the three selected journals (Virtual Reality, Frontiers in Virtual Reality, Presence: Virtual and Augmented Reality) plus, the addition of relevant literature on the language of XR reveals three major themes:

• A call for a clarification and de-colonization (an ethical expansion) of terminology used to communicate our XR experiences,
• An augmentation of our understanding of texts and of the work of their creators,
• A focus on comprehending and communicating embodiment (that is, a person’s physical experience of their given surroundings and interactions).

Notably, the majority of the articles within the three journals do not focus on issues related to how XR is communicated. They do, however, define the particular technologies and experiences analyzed in their research (augmented, mixed, or virtual reality) through their affordances (the capacities an environment provides its user: see Gibson 1977) and by using largely consistent keywords like presence, immersion, embodiment, and interactivity.

On the one hand, this linguistic consistency in definitions is both practical and useful. It allows for the creation of a body of literature around a particular topic and technology. On the other hand, it is also problematic, because what is actually meant by descriptive terms like presence, or interactivity is rarely clarified. Hence, the first theme emerges around an absence of clarity in the articles, which may lead to false presumptions, measurements, and a further entrenchment of biases. For example, it is not difficult to imagine that defining presence as the illusion of a non-mediated experience versus the willful creation of belief, would color our research focus and subsequent findings in a particular way. The former suggests that VR technology must more or less disappear in order to feel a sense of presence, whereas the latter makes no such assumptions, leaving room for more complex interpretations between the user and their technology. Authors Murphy and Skarbez (2022) home in on this definitional gap and offer clarification by unpacking this definitive term of virtuality via the question: what do we mean when we say “presence”? - see bibliographic entry below.

Other authors note that the language of platforms, including XR platforms, is largely driven by corporations (Andrejevic, 2022; Harley, 2022). Subsequently, XR technologies and applications get cloaked in a branding rhetoric of newness, which at once obscures their histories and the labor that goes into producing and sustaining them. The need to clarify our terminology and to acknowledge its roots leads to calls for a decolonization of ideologies and the grasp of the corporations often underpinning them. Specifically, authors highlight the problematic connections between virtual reality, the conquering of new frontiers, and public surveillance.

A second theme emerges around an expansion of our understanding of what we consider to be a text and the ways in which its audiences interact with it (D’Armenio, 2022; Mills, Scholes, & Brown, 2022). For example, as moving around an XR environment becomes fundamental to our experience of it, we need to consider how a user’s movement becomes a part of reading and writing an XR media text. As such, the perennial questions of media and communication studies surrounding the production and reception of meaning are reinvigorated in the context of XR.

The last theme, embodiment, is another one of the keywords used to describe XR experiences and technologies. Research shows that all human cognition and creativity is fundamentally embedded within the body and within sensorimotor processes (Gibbs, 2005). Therefore, the unique capacity for increasingly embodied experiences in XR opens up the need for a vocabulary to explain these types of interactions between texts and users. D’Armenio (2022) and Bollmer and Suddarth (2022) offer some novel terms and approaches to help us build this communicative understanding. (See annotated entries below.)

In summary, there is a call for a clarification of terminology – for instance, going one additional step of analysis beyond defining virtual reality as immersive – and for an ethical expansion of language used to communicate our XR experiences, particularly by paying attention to users’ textual embodiment as a process of meaning making. The annotated entries below expand upon each of these themes and offer some of the necessary new vocabulary to take us forward on these paths.

Calls for Clarification and Ethical Expansion of Vocabulary

Murphy, D., & Skarbez, R. (2022). What Do We Mean When We Say “Presence”?. PRESENCE: Virtual and Augmented Reality, 29, 1-43. https://doi.org/10.1162/pres_a_00360

This article, aptly published in a journal named Presence, builds on a survey that catalogues meanings and intellectual roots of the term presence (Skarbez, Brooks & Whitton, 2017). The authors examine additional literature about the term and apply philosophical and psychological lenses to unpack its most common meanings. Three common descriptions emerge: “Presence as (or as following from) “the assumption of disbelief,” presence as the “illusion of nonmediation,” and presence as “(the feeling of) being there,”” (pg. 171). The authors analyze the implicit assumptions behind each of these constructions of presence, paying particular attention to how they each connect to attention. In turn, Murphy and Skarbez identify the understandings of presence that seem the most fruitful (and those that do not), highlighting that the notion of presence is further complicated by the idea that “presence has aspects that cannot be probed or shaped by the will and, separately, aspects that can,” (pg. 172).

The authors urge those investigating presence in future studies to be specific about the (sub)definitions and assumptions of presence they subscribe to, especially those using instrument(s) to measure presence. Simultaneously, a poignant footnote cautions against the too abundant splintering of the term, which may be equally confusing. In this provocatively oscillating style, the authors raise several additional ideas. For instance, they probe whether the role of the VR user ought to be framed in a positive and agentic light, via her effortfully achieved creation of belief, as opposed to her suspension of disbelief. This points to the call for a clarification of whether the XR experiences we study function via perceptual means only or, via cognitive effort, too. It also raises questions about how we understand audiences in the context of XR.

Murphy and Skarbez also highlight the oppositional experiences of VR users who cannot stop themselves from having physical reactions (like sweaty palms) to certain experiences, like being positioned atop a thin wooden plank as it is suspended high in mid-air above a city, despite their active knowledge that what they are experiencing is not real. At the same time, they note that our very knowledge of the fact that we are using a VR system, no matter how advanced, is enough to pollute our experience, belief, and presence in it. It is here that the authors question the binary conception of presence. It either exists or it doesn’t. They instead advocate for a more nuanced position of thresholds, which must be met in order for presence to occur. What these thresholds look like and for whom is now the challenge of future research to discover.

Harley, D. (2022). “This would be sweet in VR”: On the discursive newness of virtual reality. New Media & Society. https://doi.org/10.1177/14614448221084655

This article identifies and analyzes the discursive frames surrounding corporate claims about the newness of VR. The analyses reveal several stages in one’s experience with VR, framing the initial contact as pivotal – “all it takes to believe in VR is to put on a headset,” (pg. 5) – and the ultimate goal as the conquering of new frontiers.

The author notes that the language of frontiers, pioneers, and colonization, as well as references to VR as a digital wild West, were previously identified by scholars writing about the earlier wave of VR in the late 20^th century (Chesher, 1994), and by those working on broader, critical discourses on technology and labor (Nakamura, 2020). Thus, we see a recycling of corporate language in today’s new wave of VR, which come to shape much of popular discourses on the topic.

Harley identifies another inspiration for the language, corporate ideologies, and VR workplace cultures pointing to two books: Snow Crash and Ready Player One. The former novel specifically is often credited with creating the term metaverse, although it more accurately popularized it. For Luckey (Founder of Oculus VR, which was bought by Meta in 2014 and is the world’s leader in wearable VR gear) and Abrash (Previously CTO at Oculus, now Chief Scientist at Meta’s Reality labs), these books were important communicative tools that shape their own conceptualizations of VR. This remediation and recycling of rhetoric between novels, technologies, corporate environments, and virtual reality experiences is clearly powerful but, as Harley warns, often problematic. Harley references Nakamura (2020) in the context of our current corporate branding of VR: newness, often “comes at the cost of racialized and gendered labor, problematic representations, unequal access, and a “toxic” embodiment of another person’s experiences under the guise of empathy and connection,” (pg. 3).

Andrejevic, M. (2022). Meta-Surveillance in the Digital Enclosure. Surveillance & Society, 20(4), 390-396. https://doi.org/10.24908/ss.v20i4.16008

This article gives the example of a personalized display board at an airport, aptly named a parallel reality technology by its production company, to reflect upon the increasing recession of the social, a term the author develops to describe the offloading and obfuscation of social relations to automated systems of augmented and virtual reality. Using a combination of recognition technologies, this display board can show the flight details of up to 100 passengers simultaneously thus, customizing our individual experiences of shared spaces.

The term recession of the social is inspired by Haskell’s (1977) account of the rise of professional science in the 1800s. Recession, thus, describes not the quantity of communication – this has significantly increased through the use of virtual technologies – but the “capture (and alienation) of the pattern of our communicative social fabric […] the commercially driven masking and misrecognition of irreducible forms of interdependence,” (pg. 393). Of course, Andrejevic reminds us that in contemporary societies interactive digital environments also serve as capital surveillance systems and, when filtered through these analytical lenses, our vision of sociality becomes increasingly fragmented and incapacitated.

Rethinking Texts, Authors, and the Body in XR Media

D'Armenio, E. (2022). Beyond interactivity and immersion. A kinetic reconceptualization for virtual reality and video games. New Techno Humanities. 2(22), 121-129. https://doi.org/10.1016/j.techum.2022.04.003

This article refutes the notion that virtual reality media claim a supreme capacity to induce immersion and interaction over other media. Instead, it artfully argues that the fundamental quality of virtual reality experiences is movement both, in the ways that viewers access virtual worlds by moving throughout them, and via the meaning-making that occurs through those movements. The author then proposes and illustrates the concept of a kinetic syntax – a grammar for capturing and analyzing the meaning of movement in VR.

The article notes that analytical attempts to analyze how XR experiences build meaning are still limited and labels like interactive or immersive start from the assumption of a direct correlation between increased sensory involvement and viewer engagement (Catricalà and Eugeni, 2020). Yet, the connection between the two is not a given.

Instead, the author posits that “Interactive media need to be conceived of as movement-images, meaning systems that obey a deep fusion of two syntaxes: the visual syntax, which has already been studied in semiotics and visual studies, and which pertains to the qualities of still images, and a syntax never addressed before, that is, a kinetic syntax which articulates the qualities of the movement itself,” (pg. 2).

In turn, the author proposes that such interactive experiences be understood as movement-images, a reversed and expanded formulation from Deleuze’s image-mouvement (1986).

To illustrate how a kinetic syntax might be applied, D’Armenio gives the poignant example of the old puzzle game Tetris. Tetris requires active and skilled movement a player, and which can be seen as a precursor to modern-day virtual reality gaming experiences. D’Armenio argues that the game’s increasing speed of vertically falling figures must be matched by increasingly fast and frantic movements of the player, who tries to turn the figures horizontally and to arrange them in order. Ultimately, the author argues that despite being an abstract video game, Tetris can be ‘read’ to have a universal semantic component: the human struggle against chaos, which is enacted through movement-images.

Bollmer, G., & Suddarth, A. (2022). Embodied parallelism and immersion in virtual reality gaming. Convergence, 28(2), 579-594. https://doi.org/10.1177/13548565211070691

This article examines how virtual reality and particularly, virtual gaming, produces a sense of immersion through embodied parallelism, a term the authors develop to describe “a technical mediation in which the embodied gestures and movements of a player must correspond to what is represented within a game or simulation, a correspondence which relies on, but exceeds the visual and requires strange requirements for both player (in terms of their gestures and movements) and game (in terms of including particular limits that police the movements of the player’s body),” (pg. 581).

This analytical term refuses the assumption that immersion disembodies the user in some way. Instead, immersion is “premised on an explicit engagement with – and not exclusion of – both the physicality of the body and the physicality of the medium, if in deeply contradictory ways,” (pg. 582). Embodied parallelism thus, emerges as a practical technique and analytical term for negotiating between the various material contradictions of one’s body and technical apparatus in a VR experience.

Bollmer and Suddarth exemplify embodied parallelism with virtual reality games arguing that becoming good at these games requires the player to discipline her body. They argue that in this context, immersion depends on one’s willingness to submit to a dual dynamic; on the one hand, the specific technical demands of a VR medium and on the other, one’s capacity to deliberately ignore the materiality of the mediation they are experiencing.

Throughout the article the authors draw fruitful analogies to various historical media and arts, positing that the entire history of both can be viewed either as an attempt to use various technological innovations to create a sense of immersion or, to rebel against immersivity, by creating a reflexive and critical distance from mass culture. Bollmer and Suddarth specifically draw on Wagner’s concept of Gesamtkunstwerk (total work of art), which combined poetry, music, dance, and architecture to create an overwhelming experience for his opera audiences. Notably, as part of this illusion, the orchestra and other elements of the production were hidden from sight. Similarly, VR positions the key ingredients of immersion, the body and technology, just out of view in order to be experienced ‘correctly’ in the game.

Nonetheless, the authors acknowledge that VR gaming and the constant desire to achieve a perfect, unmediated experience seem to be largely pursuits of a predominantly male, core audience. Since questions of embodiment are, tangled inseparably from psychology and physiology, it would be important for future research to explore how other populations like women or the elderly experience these sensations, too.

References

Andrejevic, M. (2022). Meta-Surveillance in the Digital Enclosure. Surveillance & Society, 20(4), 390-396. https://doi.org/10.24908/ss.v20i4.16008
Bollmer, G., & Suddarth, A. (2022). Embodied parallelism and immersion in virtual reality gaming. Convergence, 28(2), 579-594. https://doi.org/10.1177/13548565211070691
Chesher, C. 1994. “Colonizing Virtual Reality”. Cultronix 1(1), 1-27.
Eugeni, R., & Catricalà, V. (2020). Technologically Modified Self–Centred Worlds. Modes of Presence as Effects of Sense in Virtual, Augmented, Mixed and Extended Reality. In Meaning–Making in Extended Reality (pp. 63-90). Aracne Editrice.
D'Armenio, E. (2022). Beyond interactivity and immersion. A kinetic reconceptualization for virtual reality and video games. New Techno Humanities. 2(22), 121-129. https://doi.org/10.1016/j.techum.2022.04.003
Deleuze, G. (1986). Cinema: The time-image (Vol. 2). University of Minnesota Press.
Eugeni, R., & Catricalà, V. (2020). Technologically Modified Self–Centred Worlds. Modes of Presence as Effects of Sense in Virtual, Augmented, Mixed and Extended Reality. In Meaning–Making in Extended Reality (pp. 63-90). Aracne Editrice.
Gibbs R. (2005). Embodiment and cognitive science. Cambridge University Press.
Harley, D. (2022). “This would be sweet in VR”: On the discursive newness of virtual reality. New Media & Society. https://doi.org/10.1177/14614448221084655
Haskell, Thomas. 1977. The Emergence of Professional Social Science: The American Social Science Association and the Nineteenth Century Crisis of Authority. Baltimore: Johns Hopkins University Press.
Mills, K. A., Scholes, L., & Brown, A. (2022). Virtual Reality and Embodiment in Multimodal Meaning Making. Written Communication, 39(3), 335–369. https://doi.org/10.1177/07410883221083517
Murphy, D., & Skarbez, R. (2022). What Do We Mean When We Say “Presence”?. PRESENCE: Virtual and Augmented Reality, 29, 1-43. https://doi.org/10.1162/pres_a_00360
Nakamura, L. (2020). Feeling good about feeling bad: Virtuous virtual reality and the automation of racial empathy. Journal of Visual Culture, 19(1), 47-64. https://doi.org/10.1177/1470412920906259

1. It is worth noting that the majority of the articles were specifically about VR technologies. ↑