The Intel® RealSense™ SDK has been discontinued. No ongoing support or updates will be available.
Transforming the UI—Designing Tomorrow’s Interface Today (3 of 5)
Distinctions between the real and virtual worlds disappear when Intel® RealSense™ technology enriches interactive learning
By John Tyrrell
Touch and gesture control, image recognition, and augmented reality (AR) have found their way onto countless educational apps and toys, bringing virtual and real worlds together in a way that some adults struggle to comprehend. Ever more accessible and sophisticated technologies like Intel® RealSense™ technology are driving these changes, and with every new trick the devices show us, the lines become still more blurred. The result is a world of play and learning where distinctions between the real and the virtual are all but irrelevant.
3D4Medical, PlayTales, and Scholastic are among the first development teams to tackle the challenges of implementing Intel RealSense technology in the edutainment and learning field. Working closely with Intel as they experiment with the technology’s potential and overcome technical hurdles, their forthcoming apps and products will bring the next wave of interactive innovation to children and adults alike.
Interactive learning goes far beyond interactive apps for children. Ireland-based 3D4Medical creates cross-platform apps covering anatomy, health and fitness, and medical reference. Now in its fourth iteration, 3D4Medical’s Essential Anatomy* (Figure 1) is currently the top-selling medical app on Windows*, Android*, and iOS*.
Figure 1: Screenshot from Essential Anatomy
The team is currently working diligently to exploit Intel RealSense technology in ways that satisfy the stringent demands placed on apps in the medical space. Sergey Bushtyrkov, manager of software development at 3D4Medical, opted to use the Intel® RealSense™ SDK with a Unity plug-in to more easily achieve the results they were looking for while modeling limbs in the virtual space.
“The Intel® RealSense™ 3D Camera and Intel RealSense SDK provide information, but to create the end commercial application you need more than data,” explained Bushtyrkov. “We needed a 3D engine to create something attractive, so we use Unity to create an environment and manipulate the hands.”
The granularity with which individual limbs and joints are mapped takes on another level of importance in a medical context. Bushtyrkov’s team is currently focused on an anatomically accurate mapping of hands that goes beyond that required by most gesture-controlled apps. “The camera captures a certain amount of points in the hands and digitizes them,” said Bushtyrkov. “This data flow is a set of coordinates and rotation angles for each of the 22 points on the hand (Figure 2).”
Figure 2: Hand points tracked by the Intel RealSense 3D Camera
Transforming the raw data into a precisely modeled virtual hand is a complex process, and extra demands are placed on the team in terms of the detail required for a medical application. “We’re pushing the technology by creating very qualitative models of human hands, adapting them to the information that we get from the Intel RealSense 3D camera, and making them move, coordinating them,” said Bushtyrkov.
“The accuracy of the data that we’re trying to apply to hands can be a challenge,” continued Bushtyrkov. “With Intel’s help we will be able to overcome that, and we’re doing some work interpolating the hand animations using the Unity engine.”
The 3D4Medical team is in with Intel RealSense technology for the long haul, keeping with their underlying mission to always be on the sharp end of emerging technologies. “In terms of companies in our space, we’re at the forefront of technology in many different areas,” underlined Niall Johnston, vice president of business development at 3D4Medical. “We’re at the very early development stages with this technology, and it’s a great opportunity for us to research what can be done.”
Spain-based PlayTales began selling interactive books on Apple’s App Store in early 2012 and rapidly established itself as a market leader. The company has developed products based on global franchises including Garfield*, Sesame Street*, and The Little Prince, alongside its own IPs such as My New Baby*. The PlayTales storefront, with over 200 titles in seven languages, has become something of a go-to destination for parents and kids worldwide looking for great interactive storytelling.
The team is positively evangelistic about integrating Intel RealSense technology into its forthcoming Wizard of Oz* interactive pop-up book and its educational science app, Professor Garfield* (Figure 3). “Once we learned about Intel’s new technology, it was mandatory to research what we could offer customers in terms of experience, 3D graphics, and interaction,” said Enrique Tapias, CEO of PlayTales.
Figure 3: Child’s hand manipulating butterflies in Professor Garfield app
While Intel RealSense technology was in its pre-launch phase, the opportunity for PlayTales was more than one of simply implementing a new technology into its products. Instead, the process has been collaborative. “While implementing the apps, Intel was simultaneously developing the Intel RealSense SDK. We had to adapt to that,” said Pablo Benitez, CTO of PlayTales. “The good point is that we can test the new things first-hand, and we see both the app and the SDK evolving to provide a lot of cool features we can use to enhance the interactions. We really like the Contour Mode feature because it tracks the silhouette of the object closest to the camera. We use that to translate the hand of the user to our 3D world in the games, drawing a representation of the hand. With that, users can ‘collect’ water by making a bucket shape with their hands or direct butterflies to follow their fingers, for example.”
Working with Intel RealSense technology brings distinct benefits for PlayTales. “We are ahead on technology that will be in the market for years to come. That's important for us,” said Tapias. “We’re also ahead on researching the experience from the children’s side. We do our own usability testing, and we’re now getting help from Intel focus groups to test our content and the Intel RealSense technology implementation.” Benitez added, “While testing Wizard of Oz and Professor Garfield, we noticed that children had some trouble understanding the depth concept while interacting in these new 3D worlds. They also tended to move out of the camera’s field of view. As a result, we are implementing step-by-step instructions, tutorials and alerts, to teach them where they can move and how should they do it.”
PlayTales is using these insights to guide both its current and future products as consumers—and their expectations—continue to evolve at a dramatic pace. “Kids' behavior has been changing very fast and it's mainly due to the technology and devices being offered to them,” said Tapias. “I think kids have more options with Intel RealSense technology. It’s now more than just learning or reading; it’s also about interacting with content, for example, directing a butterfly’s movement in Professor Garfield.”
Tapias and his team see great potential in the new interactions that Intel RealSense technology makes possible, while recognizing that every implementation needs to be carefully controlled, polished to high standard, and appropriate for the context, whether that be learning or entertainment. For Tapias, the prognosis is distinctly upbeat: “Intel RealSense technology is definitely one of the most important pieces of technology that we can see available in the next year, and it’s not just for computers and laptops, but also for devices that kids are using, such as tablets and smartphones.”
Scholastic has a formidable reputation in children’s entertainment, ranging from books and Emmy award-winning television productions to video games and interactive learning products across multiple platforms. “We create content by marrying the subject and characters of our brands with technology,” said Jennifer Contrucci, director of digital production at Scholastic Media. “We try to use technology because it works, not as an add-on.”
Many of the company’s brands have become part of the cultural fabric of childhood in the U.S. and beyond, including Clifford the Big Red Dog* and Magic School Bus*. But it’s the company’s I Spy* brand― originally launched in 1992 as a series of children’s books and now an interactive powerhouse―that is currently receiving the Intel RealSense technology treatment.
Figure 4: Screen shot from I Spy Pirate Ship
I Spy Pirate Ship* (Figure 4) is a new addition to the brand, designed for portable all-in-one and 2-in-one computers. “The app packs in a lot of different features of the Intel RealSense SDK, using gesture, open-closed hand detection, and point tracking on the face,” said Contrucci. “You see the full spectrum of perceptual capabilities in the game.”
Kids and Cameras
One of the activities in I Spy Pirate Ship involves looking through a telescope from the crow’s nest to find objects hidden in the stars (Figure 5). When it came to implementing Intel RealSense technology to track the user’s head movements as they scan the sky, the team hit some interesting challenges.
Figure 5: Telescope view from I Spy Pirate Ship
“Because kids’ movements tend to be jerky and uncontrolled, we couldn’t consistently detect the direction they were looking,” explained Contrucci. “We changed the programming to point-tracking instead. Now the game looks for particular points on the face―the tip of the nose for example―which is proving much more effective.”
When the user finds an item, they let the program know by pausing over it, a seemingly simple interaction that really gels with the audience. “It comes off as a very magical experience for kids because they get a sense that they’re truly in the crow’s nest environment and what they’re physically doing there matters,” said Contrucci.
For Contrucci, an app’s ability to recognize when the user’s face has stopped over a specific place has significant future potential. “You can see it having many other applications,” she continued. “For example, if a young child is looking at an eBook* and she pauses over a lion picture, it’s an opportunity for the software to react and give the animal’s name and a description of it. This can give kids that ‘just-in-time’ help.”
In response to the question of whether interactivity is of genuine benefit in the educational process, Intel’s Ornit Gross, product manager of gaming and education at Intel, is unequivocal. “When you make educational material alive, immersive, and fun, it helps you remember. It helps you learn.”
“When we stepped into this domain, we learned that a range of factors can come together to enable an effective learning experience,” continued Ornit. “One of them is interactivity—whether it’s with a teacher, a friend, or an app—it needs to be interactive to deepen the learning process.”
But while Intel RealSense technology represents something of the state-of-the-art in terms of interactivity, it’s just one part of the story. “The second factor is social. You're not only learning by yourself, but you're learning in a team,” said Ornit. “That's something we feel could move further, and we’re looking at that for the next generation. It’s about how you make the experience become one-to-many instead of one-to-one, where my friend and I are learning together.”
In the meantime, Intel RealSense technology is proving its worth to the development teams taking the plunge, fundamentally changing how they—and ultimately their end-users—interact with computers and mobile devices. The voracity with which the developers have taken up the challenge is reward in itself for Ornit and her team. “What surprised me is that after the developers understood that the Intel RealSense SDK exposes both low-level data and high-level data, they coded their own interactions, taking advantage of all the SDK APIs. For example, the SDK tracks 78 landmark points of the entire face, and it detects facial expressions such as eye opening/closing and mouth openness. These capabilities were all combined so that the SDK could identify when the user made a funny face. Another example is hand-gesture recognition. The Intel RealSense SDK tracks 22 points of the hand joints as well as the pinch gesture; a developer combined those SDK APIs and identified a very unique hand pinch, which they designed especially for their game’s main character.”
As Intel knows, the technology is just the start. It’s the content that counts. Contrucci sums up Scholastic’s motivation for exploring the possibilities of Intel RealSense technology: “A lot of people think that gesture control is the next wave because of the potential to register not just what the child wants to interact with, but also that natural action. When we were taking I Spy Pirate Ship out to test that natural interaction feature with the children, it was an all-new experience for them. And that was really exciting.”
Explore Intel RealSense technology further, learn about Intel RealSense SDK for Windows, and download a Developer Kit here.
In the future, the Intel RealSense 3D Camera will be embedded into laptops, all-in-ones, and tablets. Take a peek at how faceshift used it to drive avatars on 16 tablets for the ultimate-virtual-selfie.
Intel RealSense technology makes it possible for our digital worlds to interact with our physical, organic worlds in meaningful ways. See how developer Stefan Sadchikov stepped across boundaries that just a few years ago would have been impossible to imagine.
The gaming industry is also advancing immersive experiences with Intel RealSense technology. This article provides insights from four luminary teams on the front lines of putting the technology to work.
Learn how Intel RealSense technology bridges remote and in-person communication to bring new meaning to the phrase “join the meeting.” Read the article here.
Intel and Intel RealSense are trademarks of Intel Corporation in the U.S. and/or other countries.