XR Methods and Methodologies

Katie Rawson

This analysis considers two things: 1. What research methods are currently used in three major journals dedicated to extended reality or XR (Virtual, Frontiers in Virtual Reality, Presence: Virtual and Augmented Reality), and 2. from a communication perspective, what do the ways of knowing used in these studies suggest about the field? In particular, two elements seem central to methods in XR: how we treat the body as a space for evidence and, in that vein, how we approach subjective and objective modes of evidence.

Much of the research published in the journals surveyed is experimental. Authors conducted human subjects research, which used physical measures (like heart rate and an electroencephalographic, or EEG, test to monitor reactions in the brain), subjective standardized questionnaires (like the Motion Sickness Questionnaire), or measured actions (walking, pegboard tests, performance in a test). Other conducted technical experiments that compared the performance of systems (e.g., An, 2022; Genova 2022).

Measuring the Body

Given the level of physicality that is key to experiences in XR – a physical embodiment that often surpasses previous media – it is not surprising that many of the studies focus on measuring the human body. These studies employ two approaches to making knowledge from people’s bodies: observing and combining observing with asking.

In the first kind of study, people are monitored. Studies track eye, body, or face motion; people’s position in space; or their distance from objects and encounters with them. Most of this work is aimed at either improving the mechanics of XR (e.g., W. Kim 2022, H. Liu 2022) or learning how people respond in XR (e.g., Pastel 2022, Goncalves 2022).

Second, in the combined methods approaches, researchers monitor participants’ brains and hearts to interpret how they respond to XR in combination with responses to surveys and other subjective instruments (e.g., Z. Chang 2022, Krokos 2022, Lemmens 2022, Ventura 2022). For example, Lemmens et al. (2022) combined four subjective measures, surveying attitudes and feelings, with objective measures like heartrate. One study explores the possibility of supplanting a frequently-used subjective measure for cybersickness with an EEG measurement (Krokos & Varshney, 2022). Noting the methodological challenges of either interrupting a subject or asking them to recall their experience, this study argues for using an EEG measurement for real-time feedback about cybersickness in subjects. EEGs have been effectively used in previous motion sickness studies, and this study shows promise for this method.

In these studies, the body becomes homogenized by design. Despite the fact that different people are being recorded, the studies are based on the idea of a shared set of bodily responses.

User Experience

Another significant proportion of articles are observational studies, including quasi-experimental work and user studies. The most notable trend here is the role of UX (the abbreviation for user experience from the field of design). In the XR literature broadly, there is a split between researchers conducting and describing their work as research studies and those describing their work as user studies (e.g., Pan, Alves, Jin, Risso, 2022). In part, this reflects the prominent role of design and development in the field — often the research write-up is describing a technological or platform advance. User testing is a sound approach to establishing the achievement and value of technical innovation.

However, UX and research studies have different underlying assumptions about value. Research studies are designed around knowing through verifiable practices, whether those are controlled experiments or discipline-constrained observation. UX, on the other hand, prizes individual experiences and preferences over disciplinary knowledge derived from shared methods. Users’ desires and behaviors determine the success of a user study – and the outcome, in this case, is usually to shape a technology toward users. In a field that is technologically fertile, it makes sense that we would see a balance of standard academic knowledge creation and user-focused UX studies. For social science researchers, this raises questions about how they approach a UX forward technology as they assess its social significance and, at the same time, might offer insights in methods for using XR for teaching, treatment, and research. We may also see a shift in the UX-framed studies that adopts other research methods, including mixed methods that combine scaled (e.g., surveys) or measured (e.g., EEG) responses with in-depth ethnographic observation.

We can see the ways that UX approaches are part of the XR research even in publications that are framed as more traditional scholarship. The social science research on XR often concerns effectiveness. Most of the literature measures XR techniques for effective learning (e.g., Hammar Wijkmark, Jeong, Papagiannakis) or therapy (see literature review from Weibe 2022), but there are also publications on communication and pleasure-based experience like tourism, art, and relaxation (e.g., Hall, Crolla). For many of those studies, the methods are a tried-and-true mixture of pre- and post-tests alongside questionnaires analyzed in forms that are similar to the study of other practices, treatments, or interventions.

Other Methods

Researchers are also beginning to use VR as a tool in their methods. Unlike the efficacy experiments that have VR as their subject, these are studies that use virtual environments to explore the answers to social science questions. For example, Line Tremblay, Brahim Chebbi, and Stéphane Bouchard’s research on body image uses VR and haptics, but its question is fundamentally about body image and anti-fat attitudes. It uses the affordances of VR to assess and validate theories in a controlled experiment. While much of the VR literature grapples with how to understand and create presence, this work takes the affordances of presence in order to study people in bodies.

Lastly, the articles from 2022 include a series of reviews (e.g., Atsikpasi, Hoffman, Lin, Sadowski) and literature-based theoretical models (e.g., Huang, Stohman), which speaks to the build-up of a critical mass of XR literature in a field where development is moving quickly and, at the same time, the structures of assessment and understanding are being built and debated.

It should be noted that there is a dearth of anthropological and in general qualitative methods. While there are a few ethnographic approaches to VR published in 2022 (Blackman 2022, Pike 2022), the use of participant observation, in-depth interviews, longitudinal studies, and other methods that involve human interaction and observation over extended periods of time (rather than the duration of the test or through a survey) are lacking. These are methods that seem like they could provide us with better ways of understanding and informing extended reality, given that it is a medium, as the documentary film We Met in Virtual Reality suggests, that people inhabit.

As we consider methods in XR, the content as much as the method of these works suggest that continuing to explore the body. What it means to be in a body and to use the information people get from their bodies, including for communication purposes (i.e., how we signal to each other, how we interpret signals, how we interact) are key topics for consideration. XR literature started in the fields of engineering and computer science and has been adopted in medicine and tech business but is still in the periphery of the social sciences. The methods in many of these publications reflect this, skewing toward the scientific and technical, controlled, or quasi-experiments, and not deep in qualitative or critical work. The range of methodologies to build knowledge may be expanded by continued broader engagement with XR, whether that is through using VR to test premises or figuring out how to better design and understand experiences and measures of realities.

Annotated Entries

Kroma, A., Grinyer, K., Scavarelli, A., Samimi, E., Kyian, S., & Teather, R. J. (2022). The reality of remote extended reality research: Practical case studies and taxonomy. Frontiers in Computer Science, 4. https://www.frontiersin.org/articles/10.3389/fcomp.2022.954038

Kroma et al. promote conducting more remote studies in XR and elevating best practices and standards in this form of research. Acknowledging that much XR research requires the stringent controls or the access to very specialized equipment of a lab, they describe the possibilities for research beyond those constraints. The pandemic brought with it the challenge and opportunity of doing participant-based research studies without being physically co-present. How can one do XR studies with participants in their own homes rather than in the room with the researcher? Building on principles and practices of other remote human-computer interaction and social science research, they describe six studies and classifications meant to provide a categorical guide for thinking through remote research design in XR. Their system includes five main elements: study, participants, apparatus, researcher, and data collection. Equally significant, they raise a series of issues – things that were repeated but solvable challenges in their case studies, like platform security, study protocol issues, recruitment and motivation, monitoring errors, and equipment delivery – and then research design guidelines to solve some of these issues, including workflow visuals for participants, bullet-proof and bug-free hardware and software, and research participant communities that provide social rewards (in additional to traditional incentives). Further, they argue that because remote study design makes it so that participants don’t have to come to a fixed place (e.g., don’t have to travel to a university lab) at a fixed time (e.g., sitting in a room from 3 to 5 p.m. on a Thursday), it becomes feasible for more a diverse population to join studies and for studies to be sustained over longer periods of time, which improves the research in the field.

Huang, J., & Jung, Y. (2022). Perceived authenticity of virtual characters makes the difference. Frontiers in Virtual Reality, 3. https://www.frontiersin.org/articles/10.3389/frvir.2022.1033709 Aburumman, N., Gillies, M., Ward, J. A., & Hamilton, A. F. de C. (2022). Nonverbal communication in virtual reality: Nodding as a social signal in virtual interactions. International Journal of Human-Computer Studies, 164, 102819. https://doi.org/10.1016/j.ijhcs.2022.102819

In XR literature, scholars often examine the same concept in methodologically divergent ways. Consider the approach of Junru Huang and Younbo Jung in “Perceived Authenticity of Virtual Characters Makes the Difference” (2022), as opposed to Nadine Aburumman et al in “Nonverbal communication in virtual reality: Nodding as a social signal in virtual interactions” (2022). Huang and Younbo use extant literature and argument-based methods of developing theoretical models to propose how people judge the authenticity of virtual characters. In a series of definitional and deductive steps they establish that these virtual entities are social actors defined by both their moment-to-moment agency and their representation. Their authenticity – defined by trustworthiness, originality, and spontaneity – is created through interactions. Drawing the distinction between realism and authenticity, they note that machine-identified virtual characters can be perceived as more authentic in certain contexts because authenticity is about investment and belief in the interaction that allows for imagination or alternative ways of being real (not unlike people’s relationship to fictional characters in books and movies). One of the key factors they identify in their models is the interplay of stated representation and expectations of behavior in interaction (so a character that says it is a machine and then behaves in social interaction like a machine can be more authentic than a virtual character that is hyper-real or humanlike in presentation but fails to act in a natural human way). Most of their model is based on definitional and logical assertation based in years of literature on authenticity and virtual characters with the extension of their model about communicative interactions and authenticity.

Aburumman et. al. on the other hand take up a parallel communication question: verisimilitude and the expectations surrounding social signal, in this case nonverbal communication, but employ a small study, comprising21 participants. This study used a series of tasks and questionnaires to better understand how people respond to head-nodding, eye-blinking, and facial expressions. They learn that having virtual agents engage in nonverbal communication builds trust – aligning with the theories that Huang and Jung present as well as previous studies on nonverbal communication in VR (which are legion). Neither article developed a novel methodology; however, they illustrate two key paths researchers are taking in creating knowledge in XR – the proliferation of often small, task-based studies (which is the most common approach in the literature we examined) and theoretical studies invested in framing the literature and adding to the models these studies test.

References

Aburumman, N., Gillies, M., Ward, J. A., & Hamilton, A. F. de C. (2022). Nonverbal communication in virtual reality: Nodding as a social signal in virtual interactions. International Journal of Human-Computer Studies, 164, 102819. https://doi.org/10.1016/j.ijhcs.2022.102819
Alves, J. B., Marques, B., Ferreira, C., Dias, P., & Santos, B. S. (2022). Comparing augmented reality visualization methods for assembly procedures. Virtual Reality, 26(1), 235–248. https://doi.org/10.1007/s10055-021-00557-8
An, J., Choi, G., Chun, W., Joo, Y., Park, S., & Ihm, I. (2022). Accurate and stable alignment of virtual and real spaces using consumer-grade trackers. Virtual Reality, 26(1), 125–141. https://doi.org/10.1007/s10055-021-00542-1
Atsikpasi, P., & Fokides, E. (2022). A scoping review of the educational uses of 6DoF HMDs. Virtual Reality, 26(1), 205–222. https://doi.org/10.1007/s10055-021-00556-9
Blackman, T. (2022). Virtual reality and videogames: Immersion, presence, and the performative spatiality of ‘being there’ in virtual worlds. Social & Cultural Geography, 0(0), 1–19. https://doi.org/10.1080/14649365.2022.2157041
Chang, E., Kim, H. T., & Yoo, B. (2022). Identifying physiological correlates of cybersickness using heartbeat-evoked potential analysis. Virtual Reality, 26(3), 1193–1205. https://doi.org/10.1007/s10055-021-00622-2
Crolla, K., & Goepel, G. (2022). Entering hyper-reality: “Resonance-In-Sight,” a mixed-reality art installation. Frontiers in Virtual Reality, 3. https://www.frontiersin.org/articles/10.3389/frvir.2022.1044021
Genova, C., Biffi, E., Arlati, S., Redaelli, D. F., Prini, A., Malosio, M., Corbetta, C., Davalli, A., Sacco, M., & Reni, G. (2022). A simulator for both manual and powered wheelchairs in immersive virtual reality CAVE. Virtual Reality, 26(1), 187–203. https://doi.org/10.1007/s10055-021-00547-w
Gonçalves, G., Melo, M., Barbosa, L., Vasconcelos-Raposo, J., & Bessa, M. (2022). Evaluation of the impact of different levels of self-representation and body tracking on the sense of presence and embodiment in immersive VR. Virtual Reality, 26(1), 1–14. https://doi.org/10.1007/s10055-021-00530-5
Hall, L., Paracha, S., Mitsche, N., Flint, T., Stewart, F., MacFarlane, K., Hagan-Green, G., & Dixon-Todd, Y. (2019). When Will Immersive Virtual Reality Have Its Day? Challenges to IVR Adoption in the Home as Exposed in Studies with Teenagers, Parents, and Experts. PRESENCE: Virtual and Augmented Reality, 28, 169–201. https://doi.org/10.1162/pres_a_00347
Hoffmann, S., & Mai, R. (2022). Consumer behavior in augmented shopping reality. A review, synthesis, and research agenda. Frontiers in Virtual Reality, 3. https://www.frontiersin.org/articles/10.3389/frvir.2022.961236
Huang, J., & Jung, Y. (2022). Perceived authenticity of virtual characters makes the difference. Frontiers in Virtual Reality, 3. https://www.frontiersin.org/articles/10.3389/frvir.2022.1033709
Jeong, H. S., Oh, J., Paik, M., Kim, H., Jang, S., Kim, B. S., & Kim, J.-J. (2022). Development and Feasibility Assessment of Virtual Reality-Based Relaxation Self-Training Program. Frontiers in Virtual Reality, 2. https://www.frontiersin.org/articles/10.3389/frvir.2021.722558
Jin, A., Deng, Q., & Deng, Z. (2022). A Live Speech-Driven Avatar-Mediated Three-Party Telepresence System: Design and Evaluation. PRESENCE: Virtual and Augmented Reality, 29, 113–139. https://doi.org/10.1162/pres_a_00358
Kim, W., Sung, J., & Xiong, S. (2022). Walking-in-place for omnidirectional VR locomotion using a single RGB camera. Virtual Reality, 26(1), 173–186. https://doi.org/10.1007/s10055-021-00551-0
Krokos, E., & Varshney, A. (2022). Quantifying VR cybersickness using EEG. Virtual Reality, 26(1), 77–89. https://doi.org/10.1007/s10055-021-00517-2
Kroma, A., Grinyer, K., Scavarelli, A., Samimi, E., Kyian, S., & Teather, R. J. (2022). The reality of remote extended reality research: Practical case studies and taxonomy. Frontiers in Computer Science, 4. https://www.frontiersin.org/articles/10.3389/fcomp.2022.954038
Lemmens, J. S., Simon, M., & Sumter, S. R. (2022). Fear and loathing in VR: The emotional and physiological effects of immersive games. Virtual Reality, 26(1), 223–234. https://doi.org/10.1007/s10055-021-00555-w
Lin, W., Zhu, Z., He, B., Liu, Y., Hong, W., & Liao, Z. (2022). A novel virtual reality simulation training system with haptic feedback for improving lateral ventricle puncture skill. Virtual Reality, 26(1), 399–411. https://doi.org/10.1007/s10055-021-00578-3
Liu, H., & Qin, H. (2022). Perceptual self-position estimation based on gaze tracking in virtual reality. Virtual Reality, 26(1), 269–278. https://doi.org/10.1007/s10055-021-00553-y
Pan, Z., Luo, T., Zhang, M., Cai, N., Li, Y., Miao, J., Li, Z., Pan, Z., Shen, Y., & Lu, J. (2022). MagicChem: A MR system based on needs theory for chemical experiments. Virtual Reality, 26(1), 279–294. https://doi.org/10.1007/s10055-021-00560-z
Papagiannakis, G., Kamarianakis, M., Sauter, T. C., Chalmers, A., Lasenby, J., Di Lernia, D., & Greenleaf, W. (2022). Editorial: New Virtual Reality and Spatial Computing Applications to Empower, Upskill and Reskill Medical Professionals in a Post-Pandemic Era. Frontiers in Virtual Reality, 3. https://www.frontiersin.org/articles/10.3389/frvir.2022.877494
Pastel, S., Bürger, D., Chen, C. H., Petri, K., & Witte, K. (2022). Comparison of spatial orientation skill between real and virtual environment. Virtual Reality, 26(1), 91–104. https://doi.org/10.1007/s10055-021-00539-w
Pike, S. M. (2022). Do avatars weep? Ritual and sacred space at virtual burning man. Religion, 52(2), 199–213. https://doi.org/10.1080/0048721X.2022.2051801
Risso, P., Sansone, M., & Gallace, A. (2022). Food Evaluation in Augmented Reality Environments: Can AR Affect Behavioral and Psychophysiological Responses? PRESENCE: Virtual and Augmented Reality, 29, 201–222. https://doi.org/10.1162/pres_a_00362
Sadowski, I., & Khoury, B. (2022). Nature-based mindfulness-compassion programs using virtual reality for older adults: A narrative literature review. Frontiers in Virtual Reality, 3. https://www.frontiersin.org/articles/10.3389/frvir.2022.892905
Tremblay, L., Chebbi, B., & Bouchard, S. (2022). The predictive role of body image and anti-fat attitudes on attentional bias toward body area in haptic virtual reality environment. Virtual Reality, 26(1), 333–342. https://doi.org/10.1007/s10055-021-00569-4
Ventura, S., Cebolla, A., Latorre, J., Escrivá-Martínez, T., Llorens, R., & Baños, R. (2022). The benchmark framework and exploratory study to investigate the feasibility of 360-degree video-based virtual reality to induce a full body illusion. Virtual Reality, 26(1), 323–332. https://doi.org/10.1007/s10055-021-00567-6
Wiebe, A., Kannen, K., Selaskowski, B., Mehren, A., Thöne, A.-K., Pramme, L., Blumenthal, N., Li, M., Asché, L., Jonas, S., Bey, K., Schulze, M., Steffens, M., Pensel, M. C., Guth, M., Rohlfsen, F., Ekhlas, M., Lügering, H., Fileccia, H., … Braun, N. (2022). Virtual reality in the diagnostic and therapy for mental disorders: A systematic review. Clinical Psychology Review, 98, 102213. https://doi.org/10.1016/j.cpr.2022.102213