What makes Virtual Reality powerful is that the experience is as if you were in that place. During the experience, it is your reality. That place may be real or completely virtual, i.e. computer generated. A virtual environment may represent a real place or fantasy. It might be photo-realistic or have a cartoonish level of visual realism. Any of these can lead to the experience of ‘being there,’ or not.
There is a common fallacy when evaluating VR applications that we focus heavily on appearances. Just like judging people, appearances are only skin deep. We evaluate only one facet of the experience and one that isn’t critical to having the experience of being in that other place. Visual realism, in particular photo-realism, does not automatically make a good VR experience. With photo-realism alone you might not even experience being in the other space, and I’ll explain why.
Presence
Virtual Reality researchers have for many years investigated the experience of being in another place. This is generally referred to as Presence, i.e. being present in the other place. It is really the defining characteristic of what is VR.
We know that under the right conditions, people experience VR as if it was real. VR researchers have good grasp on the necessary conditions for someone to experience presence.
Immersion is a term describing the technical requirements to induce presence. We achieve this by ‘immersing’ your senses with information that correspond to another place. The more your senses receive information only from the other place, the better. The main factors include the field of view and fully tracked head movement. Immersion is usually considered to require positional head tracking.
Why and how does Presence work?
The basic concept follows. Our brains work in two ways, which we then merge together to create our experience of the world. We have low-level (bottom-up) functions, which work from our senses to build up an understanding of what we see/hear/smell/feel. These processes are based on some very automatic (and complicated) mechanisms. The take the complex world around us and simplify it into understandable chunks. We also have high level processes that are more analytical. These align roughly with what we call our ‘thoughts’.
We like to believe that we are those high level thoughts, but the reality is much of our experience is shaped by those low level mechanisms. When we put on an HMD (VR goggles), we immerse ourselves into the world presented. VR completely covers our senses, or at least tries. This means those low-level mechanisms in our brains receive only those cues.
So what happens? As long as the basic features (edges, shapes, etc) in what we see mirrors real life, the brain doesn’t ‘see’ computer graphics, it sees objects, people, etc. For instance, a recent study showed that avatar faces are classified the same as real faces in the fast processes of the brain (Gonzalez et al. 2016).
Place Illusion
The concept of presence gets confused a bit because it actually contains two parts. What I outlined above is what Prof. Mel Slater, one of the defining researchers in Presence, later called Place Illusion. The illusion of being in another place. The low-level parts of the brain have no choice but to accept the virtual reality, if we have immersion and a world that basically mirrors reality.
Today creating Place Illusion with VR is mostly about the underlying render system. In the case of pure virtual content, the immersive requirements are supported by most engines, when combined with a HMD like the Rift or Vive. As long as developers use the existing tools, they’ve a very good chance of inducing Place Illusion. Note: 360° video is different and does not meet the higher immersion requirements, e.g. if I move my head to the side it doesn’t change the images and its not stereoscopic.
This doesn’t mean all VR works. Place Illusion can be created, yet we might not have the experience of ‘being there’. There is another component to the experience.
Plausibility
Plausibility is the second half of Slater’s updated Presence concept. It is the question: How likely is it that what I’m seeing is reality? While Place Illusion is caused automatically by low-level functions of our brain, Plausibility is higher level functions.
The experience of being there is composed of two competing processes, the automatic process based on sensory input saying “this is real” and another set of processes that might be saying it isn’t.
Visual realism is a good example. What happens when our high level functions say “this isn’t visually the same quality as I am used to seeing.” In order for our brains to deny what we are sensing, our high-level processes have to yell really loud.
Why? Because not only do our low-level functions say this is real, but that is our usual experience of the world. Unless we take certain drugs or are very sick, our senses rarely experience anything but ‘reality’. To deny what our senses are telling us about the world is very extreme, so it takes a lot to make it happen.
This creates a somewhat strange dichotomy. We can in some parts of our brain know that it isn’t real (the visual quality is low, I’m wearing an HMD, I’m not really in Rome/1700 Paris/outer-space), but the experience remains real. Many studies have shown we react very realistically, even with quite low quality virtual humans.
This is actually one of the great powers of VR, particularly for situations like treating fear. We are able to face those situations a bit easier because some part of us knows it’s not real; yet we still experience our fear reactions. This is part of why VR therapy works so well.
Breaking the Illusion
What happens when the “this isn’t real” gets loud enough? These experiences were called Break-in-Presence by Slater. At those moments, we return to our physical reality and VR becomes just images and sounds. Breaks-in-Presence tend to be fleeting, lasting only as long as those ‘voices’ remain loud.
While it is possible to enter VR and focus so entirely on the visual realism that you do not experience presence, it requires quite a lot of mental concentration. As soon as you stop focusing so intently on that one quality, the experience can start. Presence may take a short time to occur in some environments and for some people.
What causes Breaks-In-Presence and may destroy the whole experience? Technical issues and distractions from the real world are two big factors. From the content side, it is consistency. Is what I’m experiencing consistent with my normal experience of reality? Does what I’m seeing/hearing make sense together?
Visual Quality, Quality of Actions, and the Uncanny
Lets consider this in a context everyone is familiar with, films. I want you to think of your experience in the movie theater, as it is more immersive. If we are watching a modern movie the visual realism is very high. We expect then also the special effects to be of the same high quality. If they aren’t consistent, it really hurts our experience.
If we watch the first Star Wars movie today, those effects are just not acceptable, but also the visual quality is lesser. Sit a kid today down to watch it and you’re most likely to hear laughter about the bad special effects, if they even bother to continue watching.
At the other end, how do cartoons or Pixar films work? They are not visually realistic, yet immersive and enjoyable. If you saw those same actions, movements with a high visual realism you wouldn’t like it at all. Because the world is fantastic in appearance, fantastic actions are OK and even expected.
Just like a CG cartoon, a virtual environment might not be visually the same as reality, but is it consistent within itself? Avatars, in CG films and VR, are the most critical, because much of our life centers around others. Two aspects of Plausibility effect virtual humans: Do they appear and move like a human? Do they behave like a human?
Quality Mismatches
People are very good at detecting human motion and human behavior. We tend to base what we expect on visual quality, so if the visual quality is photo-realistic, we expect human movements and behavior.
What happens when this doesn’t match? It makes us very uncomfortable. This is the famous Uncanny Valley. A famous example is The Polar Express. There have been other missteps, which is why most CG films even today are cartoonish instead of photo-realism.
At the other extreme, is a VR study that Slater’s group based on the Milgram Experiments. The graphics, seen on the right, are horrible by today’s standards, as were the avatar’s movements. At the start she might have been a bad avatar, but when she started behaving like a human, she became a human. The study supports conclusion; the participants reactions aligned with the original study with human actors.
The bad graphics and movements matched reality enough for the low-level brain to create Place Illusion. They matched each other well enough as not to break the illusion. The final key was the behavior was very human and appropriate to the situation. So, it became plausible that it was reality.
Consistency not photo-realism for Presence
Proper VR automatically creates Place Illusion via immersion. Your brain takes the information provided to your senses via the HMD and headphones and places you in that world. Visual realism isn’t required for this to happen.
Whether you stay present in that world depends on the Plausibility of the virtual content. While visual-realism is a factor, it is consistency of content that matters the most. Particularly if virtual humans are in the scene, consistency is important (and hard to accomplish with photo-realism); There is a real danger of falling into the Uncanny Valley, making for a bad experience.
VR is about the experience of being in another place, not how real the graphics look. When evaluating VR keep that in mind or you might miss the best experiences.
Dr. Blom is a long time researcher in the VR field. He is the founder of Virtual Human Technologies, which applies VR and avatar technologies to human problems and helping better understand people. Virtual Orator exists largely because Dr. Blom wishes he had had such a tool instead of the ‘trail by fire’ he went through learning to speak in public.
Leave a Reply