The Case for Augmented Virtuality

AV Image

The tech world is abuzz with talk of Virtual Reality and Augmented Reality. With VR, the user can become fully immersed in a virtual world. With AR, the user sees the real world—but with digital information and virtual objects rendered in context to the real world.  You don’t hear much about Augmented Virtuality. It may be the technology to beat.

What is Augmented Virtuality?

Augmented Virtuality, or AV, is a blend of Augmented Reality and Virtual Reality. One flavor of AV has users wearing fully immersive VR headsets—but not everything has to be virtual in AV. Like Augmented Reality, the real world is part of the experience. But, instead of digital objects augmenting the real world, real world objects augment the virtual world. These real world items bleed into the virtual world as functional objects that can be interacted with in virtual experiences. One way this can be done is by adding cameras and sensors to a VR headset that can scan the real world and present it into VR. For example, such a camera could see the user’s hands, legs and feet—allowing those real world objects to become part of the virtual world.


Seeing your hands in VR

I had the chance to see my own hands using a VR headset at SXSW using the Intel® RealSense™ Smartphone Developer Kit and an Augmented Virtuality app.  This kit features an Android* phone with the Intel® RealSense™ ZR300 camera. This phone looks mostly normal—until you look at the back side. On the back edge is an array of lenses, stereoscopic IR receivers and an IR emitter. Additionally, there’s an IMU sensor and a fish eye lens to accurately track the camera position and tilt in space. This allows the device to receive object depth information from roughly 2.5-8 feet, and track that data as the camera moves. When paired with a head-mounted VR casing, the experience is very satisfying. Your hand, or anything else in the camera’s range, is immediately rendered and tracked in VR. It’s not only rendered; it becomes a fully interactive element as well. In one demo, you walk around a floating island knocking over idols, re-positioning statues, and interacting with the flow of water, all using your own hands. It’s very natural and satisfying.

AV Gallery

The VR you have may be the VR that wins
Now, you may hear people tell you that smartphone VR just isn’t capable of doing much. Mobile VR can’t render graphic detail with high frame rates like tethered VR systems. However, when you look at a smartphone enabled with depth-sensing cameras, it’s a different story. Just as mobile applications took us to a whole new space (without having the power and performance of the PC), mobile AV is following suit. The advantages of an AV-powered smartphone are:

  • Mobility: You are free to roam and experience a large space.
  • AV Technology: Your body, tools or the environment can be immediately added to the VR space.
    • Object Recognition: Defined object models can be recognized and enhanced in the virtual space with additional properties (Example: a toy rifle could fire lasers and missiles and a gem could glow and provide healing).
    • Face and Person Tracking: People around you could be recognized and/or tracked in the space, allowing you to interact with them in the virtual world.
    • Object Scanning: Any object is immediately scanned in and becomes part of the virtual world with physics properties: Furniture, walls, floors, body parts, etc.
  • Accessibility: As a phone, it’s one of the most prolific computing devices today. We just need depth-sensing cameras added to them.
  • Marketplace: Because smartphones are so prolific, there is a much larger user base and market for mobile AV. Bottom line, the VR people have may be the VR that wins.

With AV, you may not need the resolution and performance of a high end VR system. The experience makes up for it with the natural interaction of your own hand or other real world objects that can function as working elements in the VR world. There’s a lot of potential here. For example, toy manufacturers may only need to store digital models online, rather than add coded computer chips into toys so that they can be added to PC video games. If you don’t know what I’m talking about, enter a toy store. You’ll see a lot of well-crafted plastic toys that have chips so you can extend the play of that toy on a PC. But in this case, you could just let the camera see the toy, match it against the 3D model in an online library, and instantly make that toy part of the AV action.

AV: multiplayer engaged

For the multiplayer crowd, joining a game may be as simple as having a group of people in a physical room connect their phones with VR-enabled headsets. Yes, you can have real friends now. Just clean the house before you invite them over.

And finally, those childhood days of donning towel capes and using paper towel tubes as swords may be part of a future inundated with Augmented Virtuality. Those capes will turn into majestic virtual crimson cloth swaying with every movement of your body. Those cardboard tubes will turn into laser sabers and rocket launchers. Anything added to your hands will now be something that can interact with the virtual world.

I’ll leave the rest to your imagination.

Learn more about the Intel® RealSense™ Smartphone Developer Kit

For more complete information about compiler optimizations, see our Optimization Notice.