The Intel® RealSense™ SDK has been discontinued. No ongoing support or updates will be available.
Transforming the UI—Designing Tomorrow’s Interface Today (1 of 5)
Game Developers Push the Edge with Intel® RealSense™ Technology
The ways we interact with games—and how they interact with us—are in a state of constant flux as new technologies enable different experiences. And the gaming industry has always been quick to explore the potential of new interactive technologies. The industry is on the cusp of an exciting future as interfaces evolve and the new halcyon age of game development continues to deliver remarkable ideas. Playing a part in this future is Intel® RealSense™ technology, which combines state-of-the-art 3D camera sensing with advanced voice recognition built into the hardware. Using the Intel® RealSense™ SDK (beta), developers are just beginning to explore its potential to deliver immersive interactive entertainment experiences that go beyond the confines of classic hardware controllers.
This article, the first in five about the use of perceptual computing and Intel RealSense technology, provides insights from four innovative developers who are putting the technology to work.
Reshaping the Fantasy World
“We are interested in using gestural control to create a feeling of transformation and a real sense of tactile wonder in the space between you and your computer screen,” said Robin Hunicke of Funomena when describing their upcoming game that is making use of Intel RealSense technology.
Based on a belief in the positive impact of games, Hunicke co-founded Funomena in 2013. One of their aims is to explore the limits of emotional interaction between players and technology. “Our game follows a character on a journey of personal transformation and was heavily influenced by the art of origami. We love the idea of a camera that can see subtle changes in the way a player positions their hands.” High-fidelity tracking of gestures makes the input much more nuanced, and Intel RealSense technology will enable games to become truly responsive to how players move in front of the camera. “We can use that information to let them manipulate and shape objects in real time.”
While still under wraps, the game will use the precision of the Intel® RealSense™ 3D camera to read gestures, letting players connect directly with the game world and explore the game’s puzzles and environments in new ways. “We’re always interested in pushing the boundaries of what games can express. But even as we’re building something that challenges your assumptions, we’re actively embracing your input as a player. You will literally use your hands to unlock the mysteries of this world piece by piece and customize each level to reflect your vision of what the final world should look like.” said Hunicke.
Israel-based Side-Kick Games has been working with motion controllers for several years, including 2D webcams (using PointGrab*, XTR*, and EyeSight* middleware), Kinect*, and PrimeSense*. Their accumulated knowledge has made the adoption of Intel RealSense technology a straightforward process.
“We built a layer in our games for the motion controller so that we can see a lot of different middleware inputs and offer a standard motion-control interface,” explained chief operating officer, Tal Raviv. “Using that infrastructure—and our experience with motion control—the transition to Intel RealSense technology from full-body, long-range, and short-range motion control was relatively smooth. The Intel RealSense technology interface is very straightforward and it has great tracking capabilities. There were very few issues to overcome when compared to using other technologies.”
In Side-Kick’s upcoming Warrior Wave game, players use their hands to bring soldiers to safety and protect them from enemies. The team made use of different Intel RealSense technology functionalities according to the in-game context. “We use two types of controls. The first is Silhouette, which is a branch of the SDK that lets the game ‘see’ the shape of a hand but has no ‘knowledge’ of its construction (palm, fingers, thumb). Skeletal is the tracked part of the SDK that gives information about the structure of the hand (locations of each fingertip and center of the palm).” Raviv continued, “The basic game mechanics work with Silhouette, but other things such as menus work with Skeletal.”
Side-Kick Games is also ensuring that the context is appropriate for the application of Intel RealSense technology, implementing it alongside existing interface technologies in Warrior Wave to deliver an optimal experience. “Players use the touch screen to navigate through the menus and activate things because this is the most intuitive method. They use motion control while in the game and get a more immersive experience because they’re not tied to the traditional mouse and touch controls,” explained Raviv. “This combination is working best for the players now. In the future, the combination of voice and motion control could be more significant for the UI, but the main advantage of motion control today is in the gameplay itself.”
The team at Iridium Studios is using both the motion and voice capabilities of Intel RealSense technology to deliver a more realistic interface for its forthcoming real-time tactical strategy game There Came an Echo. “In the past, when directing a small squad of units in a real-time strategy game, you would use the mouse,” said Iridium founder Jason Wishnov. “But in reality you don’t draw a box around people and move them. You have to communicate with them, and you do that with your voice and gestures.”
In There Came an Echo, voice commands are vital to delivering a more realistic and immersive experience. Using accurate voice commands creates extremely engaging gameplay that more closely mimics a real-life scenario. Equally as important for Wishnov are the benefits voice recognition brings to the storytelling aspect. “We spent a lot of time and energy building the narrative and establishing these characters as actual people with real motivations, fears, and character flaws. Voice recognition helps connect the player to the characters in a narrative sense,” explained Wishnov.
Meanwhile, the motion-sensing abilities of the Intel RealSense 3D camera lend themselves to non-vocal order-giving in the context of the game’s tactical combat. “Certain military hand gestures that can map directly to in-game commands are also fun. For example, closing your fist to mean go or mark, and holding up your hand and closing it to control soldiers is very cool,” said Wishnov.
Beyond the accuracy of the technology itself, the key to making motion control work for the player is context. “If you shoehorn a set of gestures into a game just because you can, players won’t know why they're doing something and it doesn't feel cohesive,” said Wishnov. “It's very important to make sure gestures feel both relevant and natural.”
Taking a Pulse
One project that clearly exemplifies the detail and precision capabilities of the Intel RealSense 3D camera is in the biofeedback-enhanced adventure game Nevermind. In the upcoming title from Flying Mollusk, players take the role of a Neuroprober, who is able to venture into the minds of psychological trauma victims, solve puzzles, and overcome the brain’s defense mechanisms to restore their psyches. The game detects a player’s stress using biofeedback technology. When stress is detected, the environment dynamically reacts to the player’s fear and stress levels becoming more challenging and punishing as the anxiety level rises. Conversely, as the player relaxes, Nevermind becomes more gentle and forgiving. In this way, Nevermind aims to help players become more aware of, and better able to manage, their stress levels both in and outside the game world.
“I’ve always been interested in the capabilities of biofeedback and gaming,” said Erin Reynolds, creative director at Flying Mollusk. She began researching biofeedback in 2009 as a way to create a deeper connection between the player and the game. Although several biofeedback options existed, none were readily accessible for the majority of consumers, but Intel RealSense technology is changing that.
“The Intel RealSense 3D camera can detect heart rate, which is what we use to detect a player’s fear and stress. Nevermind is based on this,” said Reynolds. “Previously, when people played Nevermind, they had to wear a chest strap under their shirt. Now with the camera, they can just sit down at their computer and play. It makes a more seamless and intuitive experience.”
The Intel RealSense 3D camera’s ability to take a pulse by simply ‘looking’ at a person’s head opens up a realm of possibilities that ignites Reynold’s imagination. “It means that developers can make biofeedback a viable part of their offerings, whether it’s games, medical tools, or communication applications,” said Reynolds. “We’re beyond excited about the possibilities.”
“Another biofeedback feature of the camera is emotion-detection. Developers are very enthused about this feature, which has a lot of potential to change how games will be played in the future,” added Chuck McFadden, Intel product manager for Intel RealSense technology.
The Next Level
The use of Intel RealSense technology depends on the specific needs of the game in delivering the appropriate experience to players, as demonstrated by the different approaches of these developers. Other possible uses for Intel RealSense technology are just now emerging. “We host hackathons and game jams with our technology, traveling around the world to hotbeds of game development,” said McFadden. “We give developers the code and the Intel RealSense 3D camera, and turn them loose.”
Intel is also hosting the Intel® RealSense™ App challenge 2014, described by McFadden as a “million-dollar pot” that pushes innovators and inventors to come up with more creative ideas for the new technology. “Some ideas are things that I never would have dreamed of, such as a new way to play a musical instrument,” said McFadden. “Watching the Intel RealSense technology being taken to the next level is very exciting.”
“As developers, we're no longer confined to the interface conventions from thirty years ago,” said Reynolds, highlighting the real benefit of Intel RealSense technology. “It’s giving us a chance to look at the needs of today’s society and change the way we interact with our computers.”
“I don’t know of many people who are taking advantage of this new technology yet,” said Hunicke of the new UI possibilities just now emerging as a result of Intel RealSense technology. “We're at the cusp. We have the ability to define what those interfaces will be and what that future will look like. We can set the context, create emotional connections with users, and get them out of their comfort zone,” concluded Hunicke. “We're getting them used to the future.”
Resources and More Info
Explore Intel® RealSense™ technology and download the Intel® RealSense™ SDK for Windows Beta and Developer Kit here.
Read a geek’s perspective on why Intel RealSense technology is really important.
Soar through the air without a controller here.
Is your project ready to demonstrate? Join the Intel® Software Innovator Program. It supports developers who have forward-looking projects and provides speakership and demo opportunities.
Get started at the Developer Resource Center.