A Qualitative Post-Experience Method for Evaluating Changes in VR Presence Experience Over Time
A particular measure to evaluate an head-mounted display (HMD) based experience is the state of feeling present in virtual reality. Interruptions of a presence experience – break in presence (BIP) – appearing over time, need to be detected to assess and improve an application. Existing methods either lack in taking these BIPs into account – questionnaires – or are complex in their application and evaluation – physiological and behavioral measures –. To provide a practical approach, we propose a post-experience method in which the users reflect on their experience by drawing a line, indicating their experienced state of presence, in a paper-based drawing template. The amplitude of the drawn line represents the variation of their presence experience over time. We propose a descriptive model that describes temporal variations in the drawings by the definition of relevant points over time – e.g., putting on the HMD –, phases of the experience – e.g., transition into VR – and parameters –e.g., the transition time–. The descriptive model enables us to objectively evaluate user drawings and represent the course of the drawings by a defined set of paramters. An exploratory user study (N=30) showed that the drawings are very consistent, the method can detect all BIPs and shows good indications for representing the intensity of a BIP. With our method practitioners and researchers can accelerate the evaluation and optimization of experiences by evaluating BIPs. The possibilty to store objective paramters paves the way for automized evaluation methods and big data approaches.
The availability of consumer grade head-mounted displays (HMD) fosters the usage by a growing group of users in the research community and gives industrial users new commercial perspectives. To improve VR experiences during development and to answer research questions, methods and tools are needed to evaluate these experiences. A unique measure used for VR systems is the quantification of the user’s state to feel present in the VR.
The state of feeling present in the VR was shown to vary over time (Garau et al., 2008; Knibbe et al., 2018; Slater and Steed, 2000; Sproll et al., 2013; Steinicke et al., 2009). Temporal variations can have short-term as well as long-term effects on the user. Particularly negative effects in the course of an experience are noticed by the user, so-called breaks in presence (BIPs) (Slater and Steed, 2000). An example are people acting in the real world touching an HMD user. Also cognitive processes of the user, like getting bored or frustrated in the VR experience, can make a user attention shift from the virtual environment back to the real world (Sproll et al., 2013). Practical tools are missing that asses the presence experience or to detect these disturbances.
Today more than 90% of studies using the state of presence as a measure rely on questionnaires (Cummings and Bailenson, 2015; Hein and Mai, 2018; Rosakranse and Oh, 2014). Questionnaires are typically used post experience which can distort results due to inaccurate recall and are not able to detect variations of the presence over time (Garau et al., 2008; Liebold et al., 2017). Interviews, in addition to questionnaires, can support the evaluation of temporal events. However, interviews do not give a continuous view on the experience, require interpretation and effort by the examiner and do not give a quantification of the state of presence. Physiological measurements enable a continuous evaluation of the users’ state of being present. They are complicated to use, intrusive to a user and the association of the measured signal and the state of presence is often not clear (Garau et al., 2008; Liebold et al., 2017). Behavioral observations are not intrusive to a user. In most cases, behavioral observations do not enable continuous evaluation as they need to follow a particular protocol. This protocol might be only suitable to detect the reaction ones or only to specific points in time – e.g., an unexpected car approaching the user – (Liebold et al., 2017).
Therefore the main goal of this paper is to provide a method that enables quick, simple to use and reliable evaluation of the presence experience considering temporal variations, in particular, the detection of BIPs. To achieve this we propose a drawing method based on the work of Garau (Garau et al., 2008). The drawing method supports HMD users’ reflection on the VR experience, by drawing a continuous line along a time axis (Figure 1). The amplitude of the drawn line expresses the variations of the presence state during the cause of the experience, indicating the experience to be in a physical room or a virtual environment. As Garau’s approach is in a conceptual state it does not yet describe a full method and does not incorporate related work on temporal variations of the presence state. Hence we designed a drawing template that takes into account related work and give recommendations for its application. In an iterative design process, we optimized the template by taking into account related work and user needs while drawing. To enable quantitative evaluation and storage of the data in a database, we propose a descriptive model (Figure 3) that we derive from related work and experience from application. It replaces the users’ drawings by describing the important points and phases that occur during a VR experience, e.g., the point describing the end of the transition into VR.R (e.g., (Steinicke et al., 2009)). From the formal description we then derive concrete phases – e.g., transitioning into VR–, and further parameters describing temporal effects and changes of the presence state.
In an application example with 30 participants naive to the method, we demonstrate the feasibility and potential of the proposed drawing method. In the progress of a 15 minutes long experience, they were exposed to five breaks in presence with varying intensity. They did not know about the breaks in advance, nor where they asked to identify breaks during the experience. The resulting drawings are valid and can be used for the evaluation. We found that our method is able to detect 100% of the breaks. The detection rate will remain 100% with less participants. Further we were able to find disturbances that we did not detect with traditional usability methods during the design process of the study. Additionally there is strong evidence that the drawings reflect the strength of a break.
With our work we contribute:
A drawing template and method to collect user reports about their presence experience over time.
A model describing temporal variations of presence using parameters to replace the drawings by compact numerical data for storage, comparison and evaluation purposes.
Insights in the application of the drawing method.
As the term presence is crucial to our work, we will introduce it first. We continue by presenting literature that assesses and conceptualizes temporal variations of the presence experience.
There are multiple definitions of the term presence (e.g. (Minsky, 1980; Schubert et al., 2001; Slater and Wilbur, 1997; Witmer and Singer, 1998)). We distinguish presence from the term immersion. Immersion describes the measurable attributes of a system enabling to evoke the feeling of being present. The sense of presence is a cognitive construct that integrates incoming stimuli with user attributes, leading to a variable state of presence. The term presence is often referred to a feeling of ”being there“, the spatial presence (Schubert, 2003) or place illusion (Slater, 2009a). Other forms of presence can be experienced like involvement (Schubert, 2003) in a story or the general plausibility (Slater, 2009a)/ realness (Schubert, 2003) of an experience and social presence (Heeter, 1992). Our work is not particularly linked to one of the definitions, as the usage of the drawing method depends on the question one asks the users. The supervising person of the proposed drawing method needs to consider which definition s/he wants to follow and ask the participant the according questions. In our user study, we follow the definition of Schubert and colleagues (Schubert et al., 2001). They propose the three components realness, involvement and spatial presence that influence the presence feeling.
2.2. Assessing Temporal Variations of Presence
We focus on user self-reports. To date, self-reports offer the most practical way to evaluate an HMD-based experience, which is reflected in a high proportion of the use of the presence evaluation method (Cummings and Bailenson, 2015; Hein and Mai, 2018; Rosakranse and Oh, 2014). Continuously asking the users about their presence state during the experience is the simplest form of user self-reports. As this would foster a reorientation of a user’s attention to the real world, Slater (Slater and Steed, 2000) introduced the concept of a virtual presence counter.
The presence counter asks for the experience of breaks in presence when they appear, a binary measure, instead of reporting a presence state on a scale. Slater argues that users are not aware of the transition to VR and therefore cannot reflect the presence status in which they find themselves. Users can very well explain when they transition back into the real world, as a result from a break in presence (BIP). Although the method is convenient, it might overemphasize the report of weak BIPs. The breaks might have been unrecognized when left out for further attention, and the method is not feasible to determine the strength of a BIP (Liebold et al., 2017).
Other work used mechanical sliders to give the user the opportunity to report the feeling of being present by manipulating the position of a mechanical or virtual slider over time (Chung and Gardner, 2012; Ijsselsteijn and Ridder, de, 2000). The physicality of the sliders constantly reminds the users about the real world, although the authors tried to limit this effect, and therefore negatively influence the feeling of being present (Chung and Gardner, 2012; Ijsselsteijn and Ridder, de, 2000). The users’ ability to interact in the VR is limited, as they continuously need one hand to manipulate the slider. To avoid interruptions during the experience, we aim for a post-experience measure, which adds the possibility to evaluate temporal experiences in particular with the goal to detect and rate BIPs.
Besides the collection of data, there are a few advances to conceptualize the temporal variations of presence. Slater’s presence counter gives a stochastic view on the appearance of breaks, but does not define the temporal variation in more detail (Slater and Steed, 2000). Liebold and colleagues discuss the rationale of the strength of break in presence, based on human cognition (Liebold et al., 2017). They provide parameters to rate the degree of disruption created by a BIP for the HMD user. Their detection method relies on a physiological measure and does not give a formal description of temporal variations. Chung and colleagues describe the temporal course of a break in presence by the impact length and the recovery time from a BIP (Chung and Gardner, 2009). The impact length described by them is the drop in the presence value to the lowest value during the BIP. They do not define the parameters describing the points chosen for their measurements. They do not report how they handle plateaus in the drawings, next to or during the BIP. Garau and colleagues (Garau et al., 2008) introduced a drawing method to gain insights on temporal variations during an HMD based experience. They found that the reflection on the experience of drawing supported the users’ memory of the course of temporal fluctuations. The drawing template was not grounded in related work and therefore did not give any guidance to the user when drawing the experience. Therefore the drawings were very diverse and did not show phases of transitioning into or out of VR. As the drawings were thought to support the discussion, the presented analysis sticks to a mere description of the drawings.
Kujala and colleagues (Kujala et al., 2011), provide a method to assess user experience over several month and was positivly confirmed in other studies (Varsaluoma and Sahar, 2014; Sahar et al., 2014). In their retrospective method, the user has to draw the degree of pleasantness of using, e.g., a smartphone, being either positive or negative. They show that the method is beneficial as it supports the users’ remembering of using a phone over several month. The participants were asked to leave comments on the curve, which revealed that in the long term experience especially the hedonic quality of the system was remembered (Kujala et al., 2011), which was confirmed by Sahara and colleagues (Sahar et al., 2014). The proposed drawing method does not fit the context of a VR experience that last only several minutes or hours and is assessed immediately afterwards. Therefore the results on the application of the drawing method can not be transferred. An exemplary difference would be a better remembrance of pragmatic qualities of the system, as the time between experience and evaluation is much shorter. In addition, the question about the presence experience is different from the question of the pleasantness of a product. Different cognitive models by the participants might need a different design of the drawing template and influence the way users draw the line.
We conclude from the related work that there is a need and several attempts to asses temporal effects of a presence experience. Letting the participants draw their presence experience is a feasible approach to the problem, however a feasible method is missing. Further related work does not provide a complete model to describe the course of the temporal fluctuations. Hence we propose a descriptive model that links existing work together and provides a model of temporal fluctuations. We use the model primarily to analyze the outcome of our drawing method, but it can also be used as a communication model to discuss temporal effects of a VR experience on as it summarizes all possible effects.
3. Drawing the Presence Experience
We want to provide a method that is feasible to detect and evaluater the temporal course of presence in a VR experience. Therefore we propose a post-experience drawing method that supports the HMD user in expressing and reflecting the experience.
3.1. Design Process of the Drawing Template
To theoretical motivations we add practical findings from a collection of 60 presence drawings of four different user studies, unrelated to the study presented in this work. We collected these drawings to better understand and support participants usage of the drawing template. The participants were presented variations of the drawing template that were improved between sessions during an iterative design process. Participants were asked to draw their feeling of spatial allocation in the VR or the real world, the experienced involvement in the experience and the general plausibility of the experience. The collection of drawings was accompanied by interviews with the participants about the course of drawing and their understanding of the drawing template. The Template for the drawings included event ticks for particular tasks in a linear story-line along the time axis for about one third of the participants (similar to Figure 6). The other two thirds were not provided event ticks as the experience was exploratory without an order of events (Figure 2).
Figure 2 shows the final version of the template. As a result from the iterative design process, we added a dot that indicates the starting point for the drawing. We introduce linear, gradual transition from pure white to 25% grey from the time axis towards the extremes (See Figure 2). By this, we give the user orientation while drawing along the time axis and indicate the states in which the other world is forgotten.
3.2. The Drawing Template
Previous work uses a drawing area with a horizontal time axis and the presence state on top of it, without negative values (Chung and Gardner, 2009; Garau et al., 2008; IJsselsteijn et al., 2000). Therefore, the drawing area is limited to the bottom, which indicates to be fully present in the real world and open at the top.
We propose to provide a scale as shown in Figure 2, with the time axis indicating the middle between being fully present in the virtual and being fully present in the real world. By introducing a middle line, we provide a balanced diagram, which is equidistant to the extremes of being fully present in the real or the virtual world. Further, the middle line represents a common description by users of the transition from the real to the virtual world or going over to another world. The y-axis we chose to be 40mm in the positive and -40cm in the negative direction. The time-axis was chosen to be 200mm between the point Ptransition and the Pphysical exit (Figure 3). The end of the experience is indicated by a dashed line with a symbol below it, indicating the time of HMD removal. In Section 4 we define and discuss the defining points in more detail.
As in the ex post facto confrontation suggested by Garau (Garau et al., 2008), the remembrance of variations in presence can be supported by adding event ticks to the timeline. Ideally, these event ticks have the same order and relative distribution on the timeline as they appeared during the VR experience. The event ticks might influence how the users draw their lines as they feel the need to ”fit” the lines between the ticks. Hence, depending on the evaluation objective, the examiners must decide whether a free drawing or the support of remembrance through the presentation of event ticks is more important to them. In an exploratory VR experience, it is more difficult to provide markings of events along the timeline in advance. In this case the supervising person can manually note done some significant events during the experience and mark it on the timeline. Another approach would be to take screenshots throughout the experience and present them to the participant afterwards. In the later case the ticks and according screenshots are ideally presented on a digital drawing display as it can adapt automatically to the previous experience.
The template can be used to collect general feedback for the experience or get a detailed view on single events. In this work, we use the method as a usability tool to detect breaks in the presence experience.
First the evaluate marks descriptive points Pi (Figure 3, black points). Then all points are measured and the value is stored in a data base, e.g., a table. The position along the time-axis starts with 0mm at the line indicating putting on the HMD. The presence state is measured from the middle line with positive values towards the virtual world and negative value towards the real world, in our case 40 and -40mm.
As drawing areas might vary in their dimensions, we normalize the values collected in the previous step. Normalization is done by dividing the measured position for time by the distance between the lines indicating taking on the HMD and taking of the HMD, in our case 200mm. The presence measure is divided by the distance between the middle line and the maximum positive distance, in our case 40mm. From these values, the parameters ti and shi (Figure 3) are calculated by subtracting the presence values of the according Pbreak from Pdropping, which leads to a negative value.
For statistical analysis, we argue to treat the drawing as an ordinal scale. In particular, the presence values is a continuous scale. Therefore it is not guaranteed that the participants will use all values of the scale equidistant. For instance a change from 0.1 to 0.5 might express less impact on the users experience then the change from 0.5 to 0.6. Further one participant might experience a value of 0.6 as very high presence and another participant as low presence. The user study further down shades some light on these effects in the usage of the drawing template.
4. The Descriptive Model
Theoretical and practical insights influence our proposal of a descriptive model about temporal variations of a presence experience.
From the literature we derive the three main phases of (1) transitioning into VR (Garau et al., 2008; Oberdörfer et al., 2018; Slater and Steed, 2000; Sproll et al., 2013; Steinicke et al., 2009; Valkov and Flagge, 2017), (2) the experience itself (Garau et al., 2008; Slater, 2009b; Slater and Steed, 2000) and (3) the exit from VR (Knibbe et al., 2018). To visualize our descriptive model, we use the template introduced in the previous section (Figure 2).
In the following, we define starting and ending points Pi of the phases, phases itself and events. Points Pi have absolute values in the coordinate system. Parameters ti (time) and shi are relative values between the points Pi. The parameters shi are called shift of attention, as they describe the cognitive shift of attention (Liebold et al., 2017).
4.1. Phases of a VR Experience
The phases of every VR experience are the transition into, the experience itself and the exit out of VR (Figure 3).
Transition into VR – The transition into the virtual world can be divided into a physical and a virtual transition (Jung et al., 2018; Sproll et al., 2013). During the physical transition, the HMD user habituates with the hardware, by taking on the HMD (Jung et al., 2018; Mai and Khamis, 2018; Sproll et al., 2013). The virtual transition follows the habituation, when the user can perceive a virtual picture (Jung et al., 2018; Sproll et al., 2013). The starting point of the transition could, therefore, be defined as the contact to the VR hardware or the first moment the user can perceive a virtual picture. We argue that an HMD based experience is visually dominated (e.g., (Kohli, 2013)) and one of the critical elements in a VR experience is to maintain a stable sensomotory loop (Slater, 2009b), mainly based on the users’ visual perception. Also, Slater describes that the transition is not accompanied by a distinct perception of the user (Slater, 2009b) that could be used as a defined point in the course of an experience. Taking on the HMD is a conscious action that marks a distinct point in time, clearly identifiable for a supervising person and a participant. Habituating with the hardware will influence the transition phase (Jung et al., 2018; Mai and Khamis, 2018; Sproll et al., 2013), but does provoke an reaction in the user’s state of presence. Hence, we argue that the actual transition starts as soon as the HMD user puts on the HMD and is able to see a picture of the virtual world. Point Ptransition marks the start of transitioning (Figure 3). The end of the transitional phase is defined by any point Pi, indicating a change in the course of presence. In our exemplary Figure 3 a constant phase (green) ends the transition.
The VR Experience - The experience phase is the summary of all temporal variations and phases of constant presence while acting in VR, described in the following subsection in more detail. The first event Pi defines the start of the VR experience, Pexperience, after the transition phase. The end of the VR experience is the beginning of the last phase of dropping presence, indicated by the point Pmentalexit. We could not define a presence parameter for the experience phase. The experienced time is described by texperience (Figure 3, brown phase).
Exiting VR – The transition out of VR can be considered as the final drop in presence (Knibbe et al., 2018). The difference is that the VR experience ends after the VR exit phase. We assume that the VR exit phase follows a similar process as the transition phase but in reverse order. We base the assumption of the reverse order on Liebold (Liebold et al., 2017) description of the transition out of VR as a conscious act, which is in line with Slater description of a BIP (Slater and Steed, 2000). The transition out of VR is a sudden realization of the user that they just had focused on a virtual environment (Liebold et al., 2017). Knibbe (Knibbe et al., 2018) collected user reports about their experience of exiting VR and found the phases, first, a mental transition – getting aware that the VR experience ends – and then a physical transition – taking off the headset –. However, they also found reports by users that describe the moment of taking off the headset as the end of the experience. Therefore we assume both types of drawings to be valid. As we argue that the exit might follow a reverse order to the transition into, we add the moment of taking off the headset and being able to perceive the real world visually as a defined point to the model. The start of the VR exit phase we define as the beginning of the last phase of dropping presence which then is called Pmentalexit (Figure 3). The end of the mentalexit is defined by the end of the user’s visual immersion. The start of the Pphysicalexit is always situated on the crossing of the drawn line and the auxiliary line. At Pphysicalexit the user might not be fully present in the real world yet. By this, we take into account reports of possible aftereffects like the need for reorientation (Knibbe et al., 2018). VR exit phase ends at the point Preturn, describing the moment when participants report being fully present in the real world, again.
4.2. Events in the Progress of a VR Experience
The variation of experienced presence in a VR experience is a series of phases with constant and changing presence.
Constant Phase – Constant phases we define as phases without a change in the reported presence level. In other words, the line representing the presence state over time has a slope close to zero. We define the start of the phase of constant presence as the first point on the curve that does not show a change in presence towards the one before it. The end of a phase of constant presence we define as the last point on the curve before the slope changes from 0.
Raising Phase – The start of a phase of raising presence we define as the first point on the curve that does not show a slope. The end of the phase of constant presence can be any event Pi.
Dropping Phase – The start of a phase of dropping presence we define as the first point on the curve that shows a negative slope. The end of the phase of constant presence can be any event Pi.
Parameters are relative measures between two relevant points Pi and are represented in Figure 3 by lower case letters. The proposed concept includes known parameters, reveals new parameters and offers the opportunity to derive new measure for the presence experience, e.g., ratios.
People need different lengths of time to feel present in the virtual environment, which is represented by ttransition. Breaks in Presence (BIPs) (Slater and Steed, 2000) are represented by the Point Pbreak and the parameter shbreak. Chung (Chung and Gardner, 2009) defined a parameter similar to shbreak as the intensity of an attention shift.
The parameter tdropping helps to differentiate between sudden BIPs, e.g., a loud noise from the real world, or longer lasting effects, e.g., a tedious experience that makes the users attention drift slowly back to the real world over time (Garau et al., 2008; Liebold et al., 2017) (compare exemplary Figure 3 first and second phase of dropping presence). Further, the relation between shbreak and a following traising gives insights into the intensity of the break as recovery time was shown to be slightly correlated with the intensity of the break (Chung and Gardner, 2012). We expect, according to Slater (Slater and Steed, 2000), that tdropping in general will be shorter then traising. ttransition is expected to be longer than texit.
For the phase of exiting VR we define the three temporal parameters, total time texit, mental transition tmental and tphysical, as argued in the subsection Exiting VR above.
5. Exploration of the Method
In the previous sections we introduced a method to collect and store user reports about temporal fluctuations in the presence experience during an HMD session. As we propose, the method can be used as a method to detect breaks in presence (BIP) in an experience, a tool evaluate temporal fluctuations in more detail or a tool to compare experiences against each other. In this work, we focus on showcasing the feasibility of the method and the usage as a BIP detector. Further, this exploratory study gives insights in the usage of the drawing template and the possibilities to store and evaluate data. We conducted a controlled lab experiment to gain insights in the application of the drawing method and the descriptive model. The experience was designed in such a way that, using classical methods, it no longer showed any breaks in presence. We then introduced different kinds of BIP with varying strength. The goal of the exploration is (1) to gain insights into the repeatably of the drawing method between users and (2) the quality of detecting and evaluating BIPs.
To gather insights in the (1) repeatably we argue that the drawings need to follow a certain progress. As the drawings will have strong interpersonal variations we will look for particular events that are known from related work (see Section 4 for more detail). Hence we will analyze the drawings on key characteristics, we defined in the descriptive model (Figure 3). The characteristics we expect a user to draw are:
Lines will start at the point Ptransition.
Point Preturn exists.
Break in presence are dropping lines (shbreak).
The experience phase will cover most of the time axis.
An attention shift towards the virtual world takes a longer time span then towards the real world.
If the majority of participants follows this structure, our method is feasible of analyzing VR experiences.
To gain insights on the possibilities of the method as a BIP detector (2) we will count the detected breaks. Furthermore, we analyze possible order effects, e.g. how the early occurrence of a BIP affects the possibility to get reported. Finally, we gain insights in the possibility to reflect the strength of a BIP. Additionally, we collected qualitative feedback to gain better insights into the understanding and usage of the drawing method by the participants.
The proposed drawing method is thought to detect BIPs. We designed a linear game consisting of six subtasks to be solved by the user that did not include any BIPs. The dimensions of the virtual room were and fitted into the lab space with (Figure 4). In three iterative design steps with a total of 14 participants and four experts, we removed all unintended BIPs. We used the virtual experience test (Chertoff et al., 2010) and the igroup presence questionnaire (IPQ) (Schubert, 2003) during the design process. In addition, we conducted semi-structured interviews asking the participants for the positive and negative aspects of the experience as well as reminders of the real world and flaws in the design of the virtual world. In a final evaluation with eight novice participants the experience was rated to have a high presence on the IPQ (; ; ; ) and no events influencing the experience were reported.
During the experience, the participants fulfilled six tasks. The first was a paper-tossing, in which they threw paper balls in a garbage bin. The second was to find two pictures of cats in the room and hang it on the corresponding spot on the wall. In the third participants had to find and tidy up pens. In the fourth, the goal was to find books and put them in the right order into a bookshelf to complete the word reality. In the fifth, they had to solve a riddle which solution was to use a switch on the wall. The sixth task was also a riddle which solution was to turn on the music and make it play Beethoven.
5.3. Stimuli – Breaks in Presence
As stimuli we designed BIPs. We varied the intensity, which is a combination of violations of the users mental model about the VR and the strength of the distracting stimulus, which is described in detail by Liebold (Liebold et al., 2017). The duration was chosen to be between 2 and 4 seconds, according to related work (Chung and Gardner, 2009; Garau et al., 2008). The breaks targeted spatial presence, realness, and involvement as defined by Schubert (Schubert, 2003) and affected all senses.
We introduced five BIPs that are ordered in the following list from the strongest to the weakest:
BIP Cable Malfunction (BIPCM) A four second long blue screen error is shown in the HMD and the examiner asks the participants verbally not to move, followed by touching and moving the cable attached to the HMD to ”solve” the issue.
BIP Whitescreen (BIPWHITESCREEN) A four-second long white screen with a beeping audio track was shown similar to Garau (Garau et al., 2008).
BIP Teleport (BIPTELEPORT) The users are teleported for four seconds to an empty virtual room, having the same dimension as the main room, but showing pictures of the physical room surrounding them on the wall.
BIP Failed Interaction (BIPFAIL) Users’ had to put two framed pictures of a cat on the wall by holding it next to a marking on the wall and releasing the grasp. The second picture was made to stick after the seventh attempt to put the picture on the wall. Until the seventh attempt, the picture will fall, or the user might catch it in the air.
BIP Unrelated Controller Vibrations (BIPVIBRATION) The right controller vibrated for four seconds when it was not used for any purpose.
The breaks were triggered between 5 and 10 seconds after a task was solved. The BIPs appearance was randomized in three groups. The concept was to have a weak and strong break at the beginning and the end in the sequence of BIPs, in one condition each. Further the strong breaks were clearly separated once and one time close together. The resulting randomization orders are: A (2-4-5-3-1), B (5-4-2-1-3) and C (1-4-3-5-2).
The setup consisted of an HTC Vive HMD with according controllers111https://de.wikipedia.org/wiki/HTC_Vive , the TP-Cast system wireless system for the HTC Vive 222https://www.tpcastvr.com. The participant used Bose QC25 Noise Cancelling Headsets. The virtual experience was generated and run at 90 frames per second using Unity 5.6 on a desktop PC with NVIDIA GTX 1080 graphics card and an Intel I7 processor. The presented virtual scene was optimized to evoke presence according to the five parameters introduced by Schubert (Schubert et al., 2001). Dramatic Involvement was realized by giving the user the overall task to clean up the office, as the boss will arrive soon. Quality of Immersion was addressed by optimizing the visual appearance of the VR scene regarding 3D models, texturing and lighting. In addition, spatialized sound was used with three ambient sound sources and twenty audio sources. The controllers vibrated when an object was touched. To support Exploration of VE, all objects, but heavy furniture was movable or reactive to actions (e.g., spinning a globe on the desk, Figure 4) and behave physically to support Predictability. Interface Awareness was reduced by enabling basic interactions with just one button all the time – the users had virtual hands that supported a grasping animation –, and a wireless HMD.
30 participants (11 females, 19 males, mean age= 24) that did not take part in the iterative design process were invited by mailing lists, paper displays, social media, and personal social network. The participants had backgrounds from diverse fields. Twelve participants had never used a VR headset before, the rest reported to have used HMDs less than one time a year. None of the participants had used the drawing template before. Participants were rewarded study credits or 10 euro according to their choice.
Participants were welcomed, and the purpose of the study was explained as being a user experience study about interaction in VR. The whole study took about 60 minutes, whereby approximately 15 minutes were spent in VR. The participants were informed about anonymous storage and evaluation of their movement data given by the HTC Vive tracking system, recording of screen capture and sound in the room and asked to fill out a consent form. The usage of the HMD, noise canceling headset and controllers was explained to and tested by the participant in the standard grey Steam VR environment. As soon as the participants felt comfortable in using the system, they took off the HMD again, were guided to a starting position marked on the floor. The participants were instructed that the examiner will not help them during the experience. They were informed that the session will end after the last task and a prompt will show up, indicating the end of the experience. No information about possible BIPs was given. They started the experience on their own by taking on the HMD. The first thing the participants saw, was a transitional room. The transitional room had the same size and layout as the final room, but only a few objects were presented on a desk in front of the user. The objects were used to again train the interaction in VR. The participants were guided by instructions, which were presented textually on a monitor on the desk, during the whole experience. After completing the tutorial, users were teleported into the main room. By pressing a virtual button they activated the teleport themselves.
After the experience, the participants conducted the drawing task as described in Section 3. When finished, the examiner conducted an interview about the graphs to record details about the general course of presence and to assign the appropriate drawn BIPs to the actual ones. Finally, the participants were asked to fill out the IPQ (Schubert, 2003) and demographic information about age, sex and previous experience with HMDs.
The results section is thought to answer our research questions. Additionally we showcase how to use and analyze the data by the help of the method and the descriptive model we proposed. To get a better understanding for the method we will present the data in different levels of detail. We propose analyses that are useful to answer our research question, but future researchers are open to work with their data in their own way.
Figure 5 shows the drawing collected during the exploratory study. It showcases the diversity in user drawings. Since all the randomization groups are shown in the figure, it represents the possible result of a real application where the occurrence of BIPs is unpredictable. To obtain data that can be analyzed, we use our proposed method and the descriptive numerical model. Exemplary Figure 6 shows the marking of the points indicating the end of transitioning into and the begin of exiting VR. The values of these points are stored in the database together with the annotations describing them. We were able to store all points with the help of the proposed descriptive model and stored them the database. One Preturn could not be described by our model as the end of the line did not show a position that is defined as being fully present in the real world and therefore was excluded.
Tabel 1 presents the outcome of a statistical evaluation of the drawings based on the database for the single randomization groups. It shows the evaluation of points (Pbreak) as well es derived parameters (shbreak). Figure 7 is en example for the visualization of the numerical data. It shows Box-Plots for the introduced BIPs, combining the results of all randomization groups.
5.7.1. The Transitional Phases
Figure 6 shows the drawn transition and exit phases, without the experience in between. Six participants experienced a disturbance during transition, not caused by the BIPs we introduced intentionally. We exclude them in Figure 6 (left picture). The unintended disturbances are caused by the HMD not fitting and readjustments by the participants. One very shallow transition drawing was caused by boredom during the instructional part, leading to a low feeling of involvement. The last one did not give a justification. Exiting VR was influenced for one participant of Randomization C as the BIPWHITESCREEN happened as the last break before the end of the experience. It was removed in Figure 6 (right picture) as no exit phase was drawn according to our definition. The excluded cases are still visible in the overview of drawings (Figure 3).
Further, Figure 6 is an example for the application of the analysis (Figure 1). As we are able to store the data in a database, we can calculate parameters, for instance on the experience of time. In average participants used () of the timeline to report the transition into VR (ttransition). Further they used () to express the transition out of VR (texit). As a result the experience phase took 71% of the available space. Ptransition was always situated at the defined position, as it was an instruction. Participants finished their drawings in average at Preturn () of the timeline, and at () of the negative presence axis.
In no cases, participants reported separating their exit experience in the mental and physical exit. 25 of the drawings show an drop which accelerates continuously over time and shows a uniformly accelerated slope of the curve between the points Pmentalexit and Preturn. Five participants drew their experience as a sudden breakpoint at which Pmentalexit is at the same value on the timeline as Preturn.
5.7.2. Breaks in Presence
Detection Frequency – The detection frequency is the percentage of participants reporting a BIP. We report the detection frequencies in Table 1. Strong BIPs – Cable Malfunction, White Screen – are more likely to be reported than weak BIPs, no matter if they appear early or late during the experience. BIPFAIL stands out from this with a high detection rate in two randomization groups. Although, we defined it as a weak BIP and the disturbance resulting from it was perceived by the participants as such during the experiment (Figure 7), the detection rate was similar to the stronger BIPs. An individual participant was able to identify 2.8 BIPs () on average and across all randomization groups.
Order effects – The order might show an effect in our study, but the results are not clear as the number of detections was low for some BIPs, which prevents further statistics. The two strongest breaks are 10% and 40% more likely to be reported when in the last position of the BIPs compared to being the first. However the weaker breaks do not show such an effect. The BIP failed interaction, which is in the same position of the sequence of BIPs in all randomizations, shows a variation in the detection rate between 50% to 90%. The unrelated vibration of the controller was not reported, when in the first position of the randomization – Position 1, Randomization B –.
Intensity – Figure 7 shows Box-Plots for the presence value of Pbreak considering all randomization groups as one. The intensity for Pbreak in the single groups can be found in Table 1 but is not used for further discussion, as the number of participants is to small to recognize the trend. A trend in the order of the BIPs’ strength is visible that fits the prediction we derived from related work in Section 5.3. The weaker breaks are drawn with a tendency of their median towards the virtual world. In particular, the strongest BIP BIPCM is reported to be close to the real world, the weakest BIP BIPVIBRATION. We did not find a tendency for the parameter shbreak in our study.
Time Parameters – At this point we want to stress that the timely order is not of particular interest for the detection of BIPs and in our study. However, it gives an estimate on how precise users are able to report the appearance of a BIP, if the method is used in a real world application. We were interested in the ability to report appearance of BIPs relative to events in the experience. We counted the correct positioning of Pdropping or the following Pi. BIPs were counted as positioned correctly if one of the points was situated within 5mm (2.5%) before and 25mm (12.5%) after the according task tick. We found that participants had 97 of the 118 detected breaks positioned correctly. As a result in 18% of the cases the relation between the event in VR and the drawing would not be clear.
We further compared the time it takes that a BIP comes into full effect, tdropping, to the time participants need to recover from it, traising. We calculated the average of each and then divided the result of tdropping by traising. The resulting ratio is 0.8, which means that a dropping phases is drawn 20% shorter than a raising phase.
5.8. Other Anecdotal Findings
When drawing the curves, many users started to reflect on the drawings, stimulated by the drawn line. Some recognized, after drawing a second BIP, that previous one was stronger. Therefore they went back to the first and changed the curve to fit relative to the second. We found similar behavior for ordering the appearance BIPs and their relative position to the event ticks on the timeline.
The method enables users to reflect temporal variations – The drawings fulfill our five prerequisites (Section 5.1) to report temporal variations during a presence experience. In particular we could show that all participants started at Ptransition (prerequisite a) and end with Preturn (prerequisite b). Only one event was drawn that did not fit into the model. In the particular case, presence was reported as raising again after the last dropping phase (Figure 3, randomization C). All participants understood that a BIP is an attention shift towards the real world and drew it as a dropping line (prerequisite c). Most of the template (71%) was used to express the temporal variation during the main experience phase (prerequisite d). According to the related work, we showed that participants report an attention shift towards the real world to happen faster than towards the virtual world (prerequisite e). In particular ttransition was 38% longer than texit and to recover from a break (traising) took in average 20% longer than the break (tdropping).
In summary, we were able to show that our method is superior to established methods, e.g., by using Garau’s approach (Garau et al., 2008), when it comes to understanding temporal variations in the presence experience. The expression was not possible in such possible before. We want to stress at this point, that the participants had minor experience in HMD usage and were not given any information about expectations on the drawings progress. The strong coherence between the theoretical model we provide in section 5.1 and the drawings, reflecting the users actual mental model, underlines the feasibility of our method.
Drawing Method is a tool to detect BIPs – The following paragraphs target research question two that we raised in Section 5. The groups of ten Participants were able to detect all BIPs, but the weakest BIP when it occurred at the beginning of the experience (Table 1). The two strongest breaks base on visual disturbances and are detected by at least 60% of the participants. We find it interesting that the failed interaction was reported by 50% to 90% of the users. A failed interaction is not a technical failure disturbing the perception of the virtual world. It is a logical failure of the experience, disturbing the plausibility illusion (Slater, 2009b) for the HMD user.
Furthermore, we found BIPs that we did not find during the design phase with established methods. In particular, in the transition phases we identified 7 disturbances, with accompanying user reports on the effects on the further course of experience. These results underline the possibilities offered by our method compared to established methods. Additionally, it demonstrates the advantage of being able to analyze the temporal cause of the presence experience. The disturbance in the beginning affected the user over a long period in time, which would have been impossible to express with a questionnaire and more complex to explain in an interview.
Similar to other post-experience methods the drawing method suffers from inaccurate recall by the participants (Garau et al., 2008). However, in each randomization group, with 10 participants each, almost all breaks were reported (Table 1) by at least 20%. As a result, we argue that 10 participants are enough to discover all BIP in a VR experience with a maximum duration of 15 minutes. There might be a chance to need even fewer participants, but with the risk of not finding a weak disturbance that occurs early in an experience. We want to highlight at this point, that using 10 or less people to find errors in a system is similar to established recommendations from usability testing of other systems (Nielsen and Landauer, 1993). Longer experiences might demand more participants, to counteract inaccurate recall of weak breaks at the beginning of the experience.
In summary, our method is able to detect a variety of BIPs belonging to all aspects of the presence experience. Finding BIPs that we have not consciously implemented in the experience shows the advantage of our method over established methods.
Drawings indicate the intensity of a BIP – The users reported the disturbance introduced by the BIPs (Figure 7). The points Pbreaks represent BIPs the best, which is in line we Chung’s suggestion (Chung and Gardner, 2012). We were speculating that the parameter shbreak might report the relative evaluation by the participants. We could not find a clear trend for shbreak, as it might depend on things that precede or follow. However, future research might have a closer look on the parameter shbreak, as it has the potential to analyze accumulative effects of events on the overall presence experience.
In contrast to the overall intensity for PBreak (Figure 7), the single randomization groups do not show a clear trend for the intensity of a break (Table 1). One reason might be that the lower number of participants is not enough to detect a difference. Another be that the randomization levels out effects that are introduced by the order the breaks had in our study.
From this, we conclude that our drawing template has good potential to evaluate the intensity of a BIP. However, the number of participants needed to evaluate the strength of a break needs to be much higher then for the detection of the breaks. It needs to be mentioned that we decided not to remove outliers. We decided to leave all data in the database, as the users decided to draw the line according to their believe on how to use the model. Therefore they did nothing wrong that justifies an exclusion. It will need further studies to gain a deeper understanding of why people chose to draw their line very differently to others to decide for outliers.
Drawing the transitional phases has potential – We could show that people report the experience of a transition phase in average around three times longer than the exit phase (Figure 6). This is according to the assumption that people do harder to detect the change of the presence state towards the VR but the break shifting their attention towards the real world (Liebold et al., 2017; Slater and Steed, 2000).
We could not find a separation of the mental and physical exit of VR (Knibbe et al., 2018). Novice participants might be overwhelmed by the things happening in the short time during the exit. 83% of the participant drew the exit as a continuous curve that might indicate a separation of these phases. The rest drew a sudden break (Figure 6). However the continuous curves might be a result of a more natural or comfortable drawing behavior and not related to the experienced changes. Informing the user about possible effects during the exit phase might support a more differentiated report.
As our study was not designed to evaluate the transitional phases, certain parameters might need clarification in future studies. For example, we defined the parameter ttransition to be the time between Ptransition and the next prominent point, as it is an ongoing process. However, the time ttransition it actually needs to develop a sufficient presence experience might be indicated by the point that crosses the middle line between real world and virtual world.
We can not answer these questions here, as they are not the main goal of hour study. However, in conjunction with the findings discussed above about detailed user reports on their transitional phases, we see great potential to better understand the transitional phases of a VR experience when using our method.
The report of a longer transition time into VR then to exit VR, is in line with the related work. However, there might be a bias introduced by the drawing template. In the template the ticks for the tasks – e.g., putting a picture on the wall – are distributed along the time axis according to their appearance relative to events in the experience. The first task in the experience – paper toss – was much later in time, then it took between the last task – Riddle Beethoven – and the end of the experience. Therefore, there was much more space to draw the transition into than the exit of VR, which might lead to the current result (Figure 6).
We do not cover drawings that show a step raising at the beginning and then change to a more shallow curve when transitioning into VR. We decided to stick to our decision and used the first point of constant presence or dropping presence for the parameters ttransition and shtransition. However, people might express that they felt present in the virtual world very fast, but became more and more confident over time, as known from other studies (Mai et al., 2018).
The experience in this study took about 15 minutes. Longer experience might amplify possible inaccurate recall of the experience. An analysis of longer experiences should include more than 10 participants.
7. Future Work
The current work aimed on creating the drawing method and the proposal of a descriptive model to describe the outcome in an objective way. Exemplary we chose the use case to explore the method was the BIP detection. Motivated by the demonstrated the feasibility, we argue to explore other forms of using the template, e.g., to compare experiences to each other. For instance, specific questions, like the comparison of transitional environments, we suggest to present the same drawing template after each condition. The users then can reflect on the drawn line of the previous condition and express the latest experience by drawing the line relative to the other. In this way, we expect more detailed insights into the temporal course of a presence experience, e.g., influenced by system design or story telling. We found a clear tendency that the method is used to express the strength of a BIP. Future work, therefore, might work on the elaborate the method and test its validity and reliability to transform it into a measurement tool, as questioannaires are today.
In contrast to the established questionnaires – main measurement method in more than 90% of studies –, the usage of the drawing method gives insights into temporal effects. Methods enabling the assessment of temporal variations, like physiological and behavioral measures, are hardly not used.
The method we provide enables users to express variations in their presence experience over time by drawing. To enable the evaluation of the drawings, we propose a descriptive numerical model based on related works in presence research. Our descriptive model replaces the drawings by compact numerical data for storage, comparison and evaluation purposes.
The drawing method is applied after an experience and allows the participant to draw a line that indicates the state of presence in VR over time by changing the amplitude of the line during drawing. In an exploratory user study, we demonstrate the feasibility of the method. We show the higher accuracy in understanding temporal variations of the presence experience compared to existing methods. In particular, we could show that our method is feasible for detecting 100% of breaks in the presence experience (BIP) requiring only 10 participants. Participants self reflection on their experience was stimulated by the drawing, resulting in a very detailed expression of their presence experience. Additionally, participants were able to express the influence of a BIP on the following cause of the experience by their line drawing. In addition, we have good indications that, by collecting data from more participants, the method can be used to evaluate the strength of a single break in attendance. The proposed drawing method in combination with the descriptive model closes the gap between the existing cumbersome techniques and the questionnaires that do not capture temporal variances.
- Chertoff et al. (2010) Dustin B. Chertoff, Brian Goldiez, and Joseph J. LaViola. 2010. Virtual Experience Test: A Virtual Environment Evaluation Questionnaire. In Proceedings of the 2010 IEEE Virtual Reality Conference (VR ’10). IEEE Computer Society, Washington, DC, USA, 103–110. https://doi.org/10.1109/VR.2010.5444804
- Chung and Gardner (2009) Jaeyong Chung and Henry Gardner. 2009. Measuring Temporal Variation in Presence During Game Playing. In Proceedings of the 8th International Conference on Virtual Reality Continuum and Its Applications in Industry (VRCAI ’09). ACM, New York, NY, USA, 163–168. https://doi.org/10.1145/1670252.1670287
- Chung and Gardner (2012) Jaeyong Chung and Henry Gardner. 2012. Temporal Presence Variation in Immersive Computer Games. International Journal of Human-Computer Interaction 28 (08 2012), 511–529. https://doi.org/10.1080/10447318.2011.627298
- Cummings and Bailenson (2015) James Cummings and Jeremy Bailenson. 2015. How Immersive Is Enough? A Meta–Analysis of the Effect of Immersive Technology on User Presence. Media Psychology 19, 2 (2015), 272–309. https://doi.org/10.1080/15213269.2015.1015740
- Garau et al. (2008) Maia Garau, Doron Friedman, Hila Ritter Widenfeld, Angus Antley, Andrea Brogni, and Mel Slater. 2008. Temporal and Spatial Variations in Presence: Qualitative Analysis of Interviews from an Experiment on Breaks in Presence. Presence: Teleoper. Virtual Environ. 17, 3 (2008), 293–309. https://doi.org/10.1162/pres.17.3.293
- Heeter (1992) Carrie Heeter. 1992. Being There: The Subjective Experience of Presence. Presence: Teleoperators and Virtual Environments 1, 2 (1992), 262–271. https://doi.org/10.1162/pres.19220.127.116.112 arXiv:https://doi.org/10.1162/pres.1918.104.22.1682
- Hein and Mai (2018) Dimitri Hein and Christian Mai. 2018. The Usage of Presence Measurements in Research: A Review. In Proceedings of the International Society for Presence Research Annual Conference (Presence). The International Society for Presence Research.
- Ijsselsteijn and Ridder, de (2000) Wijnand Ijsselsteijn and Huib. Ridder, de. 2000. Measuring temporal variations in presence. In Papers from the workshop on presence in shared virtual environments, Ipswich, UK, June 10-11 1998.
- IJsselsteijn et al. (2000) Wijnand IJsselsteijn, Huib Ridder, de, Jonathan Freeman, and Steve Avons. 2000. Presence : concept, determinants and measurement. In Human Vision and Electronic Imaging V, January 24-27, 2000, San Jose, USA (Proceedings of SPIE - The International Society for Optical Engineering), B.E. Rogowitz and T.N. Pappas (Eds.). SPIE, United States, 520–529.
- Jung et al. (2018) Sungchul Jung, Pamela J. Wisniewski, and Charles E. Hughes. 2018. In Limbo: The Effect of Gradual Visual Transition Between Real and Virtual on Virtual Body Ownership Illusion and Presence. In 2018 IEEE Conference on Virtual Reality and 3D User Interfaces, VR 2018, Tuebingen/Reutlingen, Germany, 18-22 March 2018. 267–272. https://doi.org/10.1109/VR.2018.8447562
- Knibbe et al. (2018) Jarrod Knibbe, Jonas Schjerlund, Mathias Petraeus, and Kasper Hornbæk. 2018. The Dream is Collapsing: The Experience of Exiting VR. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI ’18). ACM, New York, NY, USA, Article 483, 13 pages. https://doi.org/10.1145/3173574.3174057
- Kohli (2013) Luv Kohli. 2013. Redirected Touching. Ph.D. Dissertation. Chapel Hill, NC, USA. Advisor(s) Brooks,Jr., Frederick P. AAI3562754.
- Kujala et al. (2011) Sari Kujala, Virpi Roto, Kaisa Väänänen-Vainio-Mattila, Evangelos Karapanos, and Arto Sinnelä. 2011. UX Curve: A Method for Evaluating Long-term User Experience. Interact. Comput. 23, 5 (Sept. 2011), 473–483. https://doi.org/10.1016/j.intcom.2011.06.005
- Liebold et al. (2017) Benny Liebold, Michael Brill, Daniel Pietschmann, Frank Schwab, and Peter Ohler. 2017. Continuous Measurement of Breaks in Presence: Psychophysiology and Orienting Responses. Media Psychology 20, 3 (2017), 477–501. https://doi.org/10.1080/15213269.2016.1206829 arXiv:https://doi.org/10.1080/15213269.2016.1206829
- Mai and Khamis (2018) Christian Mai and Mohamed Khamis. 2018. Public HMDs: Modeling and Understanding User Behavior around Public Head-Mounted Displays. In Proceedings of the 7th ACM International Symposium on Pervasive Displays (PerDis ’18). ACM, New York, NY, USA. https://doi.org/10.1145/3205873.3205879
- Mai et al. (2018) Christian Mai, Tim Wiltzius, Florian Alt, and Heinrich Hußmann. 2018. Feeling Alone in Public. Investigating the Influence of Spatial Layout on Users’ VR Experience. In Proceedings of the 10th Nordic Conference on Human-Computer Interaction (NordiCHI ’18). ACM, New York, NY, USA. https://doi.org/10.1145/3240167.3240200
- Minsky (1980) Marvin Minsky. 1980. Telepresence. Omni 2, 9 (1980), 45–52.
- Nielsen and Landauer (1993) Jakob Nielsen and Thomas K. Landauer. 1993. A Mathematical Model of the Finding of Usability Problems. In Proceedings of the INTERACT ’93 and CHI ’93 Conference on Human Factors in Computing Systems (CHI ’93). ACM, New York, NY, USA, 206–213. https://doi.org/10.1145/169059.169166
- Oberdörfer et al. (2018) Sebastian Oberdörfer, Martin Fischbach, and Marc Erich Latoschik. 2018. Effects of VE Transition Techniques on Presence, Illusion of Virtual Body Ownership, Efficiency, and Naturalness. In Proceedings of the Symposium on Spatial User Interaction, SUI 2018, Berlin, Germany, October 13-14, 2018. 89–99. https://doi.org/10.1145/3267782.3267787
- Rosakranse and Oh (2014) Christine Rosakranse and Soo Youn Oh. 2014. Measuring Presence: The Use Trends of Five Canonical Presence Questionaires from 1998–2012. In In Proceedings of the 15th International Workshop on Presence (ISPRâ14).
- Sahar et al. (2014) Farrukh Sahar, Jari Varsaluoma, and Sari Kujala. 2014. Comparing the Effectiveness of Electronic Diary and UX Curve Methods in Multi-component Product Study. In Proceedings of the 18th International Academic MindTrek Conference: Media Business, Management, Content & Services (AcademicMindTrek ’14). ACM, New York, NY, USA, 93–100. https://doi.org/10.1145/2676467.2676500
- Schubert (2003) Thomas Schubert. 2003. The sense of presence in virtual environments: A three-component scale measuring spatial presence, involvement, and realness. Zeitschrift für Medienpsychologie 15, 2 (2003), 69–71. https://doi.org/10.1026//1617-6322.214.171.124
- Schubert et al. (2001) Thomas Schubert, Frank Friedmann, and Holger Regenbrecht. 2001. The Experience of Presence: Factor Analytic Insights. Presence: Teleoperators and Virtual Environments 10, 3 (June 2001), 266–281. https://doi.org/10.1162/105474601300343603
- Slater (2009a) Mel Slater. 2009a. Place illusion and plausibility can lead to realistic behaviour in immersive virtual environments. Philosophical Transactions of the Royal Society of London B: Biological Sciences 364, 1535 (2009), 3549–3557. https://doi.org/10.1098/rstb.2009.0138 arXiv:http://rstb.royalsocietypublishing.org/content/364/1535/3549.full.pdf
- Slater (2009b) Mel Slater. 2009b. Place illusion and plausibility can lead to realistic behaviour in immersive virtual environments. Philosophical Transactions of the Royal Society of London B: Biological Sciences 364, 1535 (2009), 3549–3557. https://doi.org/10.1098/rstb.2009.0138 arXiv:http://rstb.royalsocietypublishing.org/content/364/1535/3549.full.pdf
- Slater and Steed (2000) Mel Slater and Anthony Steed. 2000. A Virtual Presence Counter. Presence: Teleoperators and Virtual Environments 9, 5 (Oct. 2000), 413–434. https://doi.org/10.1162/105474600566925
- Slater and Wilbur (1997) Mel Slater and Sylvia Wilbur. 1997. A Framework for Immersive Virtual Environments (FIVE): Speculations on the Role of Presence in Virtual Environments. Presence: Teleoperators and Virtual Environments 6, 6 (1997), 603–616. https://doi.org/10.1162/pres.19126.96.36.1993 arXiv:https://doi.org/10.1162/pres.19188.8.131.523
- Sproll et al. (2013) Daniel Sproll, Jacob Freiberg, Timofey Grechkin, and Bernhard Riecke. 2013. Poster: Paving the way into virtual reality - A transition in five stages. In IEEE Symposium on 3D User Interface 2013 - Proceedings (SUI ’13). 175–176.
- Steinicke et al. (2009) Frank Steinicke, Gerd Bruder, Klaus Hinrichs, Anthony Steed, and Alexander Gerlach. 2009. Does a Gradual Transition to the Virtual World increase Presence?. In 2009 IEEE Virtual Reality Conference. 203–210. https://doi.org/10.1109/VR.2009.4811024
- Valkov and Flagge (2017) Dimitar Valkov and Steffen Flagge. 2017. Smooth Immersion: The Benefits of Making the Transition to Virtual Environments a Continuous Process. In Proceedings of the 5th Symposium on Spatial User Interaction (SUI ’17). ACM, New York, NY, USA, 12–19. https://doi.org/10.1145/3131277.3132183
- Varsaluoma and Sahar (2014) Jari Varsaluoma and Farrukh Sahar. 2014. Measuring Retrospective User Experience of Non-powered Hand Tools: An Exploratory Remote Study with UX Curve. In Proceedings of the 18th International Academic MindTrek Conference: Media Business, Management, Content & Services (AcademicMindTrek ’14). ACM, New York, NY, USA, 40–47. https://doi.org/10.1145/2676467.2676485
- Witmer and Singer (1998) Bob G. Witmer and Michael J. Singer. 1998. Measuring Presence in Virtual Environments: A Presence Questionnaire. Presence: Teleoper. Virtual Environ. 7, 3 (June 1998), 225–240. https://doi.org/10.1162/105474698565686