Archived - Facial, Gesture, and Voice Recognition Go Social with Finding BBB

The Intel® RealSense™ SDK has been discontinued. No ongoing support or updates will be available.

Download Document

By John Tyrrell


Finding BBB* is an Intel® Perceptual Computing Challenge award-winning game created by TheBestSync, a development studio based in Guangzhou, China. Finding BBB combines the functionality of Intel® RealSense™ technology capabilities—including facial, voice, and gesture recognition—with Facebook* connectivity, personalization, player-created missions, and location-based services. This combination of elements delivers what TheBestSync believes to be an innovative and intuitive user experience.

Figure 1: Players use gesture controls to navigate their bee through the 3D environment.

Finding BBB is a freemium social interactive game that builds on the studio’s previous titles BBB GOAL* and JOY*. JOY was a Phase 1 Grand Prize winner in the Intel® Perceptual Computing Challenge. While the game focuses on implementation of the Intel® RealSense™ SDK UI, TheBestSync team has also enabled traditional mouse and keyboard controls and a touch interface for two reasons: first, because gesture and voice recognition are not yet widely used in the gaming industry and are unfamiliar to many players; and second, because it’s important to give players a choice. Players can also use multiple input modalities in parallel, giving them ample freedom in how they play the game.

Figure 2: In addition to gesture, the player can use keyboard controls if desired.

Optimizations and Challenges

Gesture Control

One of the most significant parts of the optimization work that the team undertook was related to interpreting the data provided by the Intel RealSense SDK. They discovered that the extremely high level of responsiveness and accuracy of the Intel® RealSense™ 3D camera—and the precision and speed of the resulting data—created an unexpected situation: The slightest natural movement of the hand was being translated into sometimes unnecessary movements of the game character. To curtail these extra movements, the team coded a filter that limited the sampling rate on the data stream and thus smoothed the hand movements that the Intel RealSense SDK delivered to the game. This ultimately stabilized the characters’ on-screen movements to a point that was acceptable to the player and visually satisfying in the context of the game environment.

Figure 3: Players are shown step-by-step how to use the gesture controls.

Through the testing process, the team was also surprised to discover that the system they implemented for controlling the bee character in-game initially caused some difficulties for players who were unaccustomed to the combination of a 3D environment and gesture control. The team attributed this to players being more accustomed to using 2D mouse controls where only the x- and y-axes are considered. The addition of the z-axis in the context of the full 3D gesture control interface caused issues for some players in establishing their position, moving backward or forward, turning around, or otherwise navigating the 3D environment.

To improve the user experience, the team worked on reducing the learning curve for the controls. To do this, they implemented an in-game, step-by-step immersion process into the 3D control scheme. The first stages of the game are played using 2D controls, allowing players to become accustomed to the basic gesture controls before being introduced to the additional z-axis. These handholding stages proved useful in helping players master the controls and have fun with the game.

Location-Based Services

TheBestSync also enhanced the immersive, augmented reality of Finding BBB by recreating real-world weather and day and night cycles in the game. By integrating the OpenWeatherMap API, which connects to the game using JavaScript*, location-specific data on weather and the time of day is acquired for every player. These conditions are then reflected in the game universe in real time. If it’s raining, cloudy, or nighttime in the player’s vicinity, Finding BBB will show this in-game.

Figure 4: Real-world atmospheric conditions such as rain are rendered in the game world.

Voice Recognition

Intel RealSense SDK voice recognition is used for the spelling tasks in the game when the player must say words out loud to complete missions. The game contains a number of preset words (for example, “NAME”) the letters of which are displayed jumbled up on-screen. The player then needs to work out what word the letters spell and say it out loud. The Intel RealSense SDK voice recognition capability then detects whether the player said the correct word.

Players also have the option to set their own similar spelling tasks and share them with friends.

Figure 5: Players complete spelling challenges using the voice-recognition functionality.

Facial Recognition

In Finding BBB, players take photos of themselves and map their images onto the bee character that they control in the game universe. TheBestSync views this integration of the player into the game world as a form of augmented reality, a field in which the company has many roots. The ultimate goal of Finding BBB is to make the player into the hero of the game.

Figure 6: Facial recognition is used to map the player’s face onto their bee character.

Social Functionality

Because the social relationships between players are important to the game experience, Finding BBB uses the Facebook API, employing its social login and each player’s Facebook friends list. However, in China, where Facebook is blocked, the game uses the similar functionality of the WeChat* and Viber* social networks. Because of the inability to directly access Facebook locally, the team used a VPN in order to have a non-Chinese IP address to test the Facebook functionality of the game. Although this can result in a lower connection speed than the team would achieve if they were able to connect locally, using a VPN is a simple solution that can be deployed to address this issue.

Figure 7: The Facebook* social login allows players to connect and share with friends online.

Testing and Analysis Using the Intel® VTune™ Amplifier

The testing of Finding BBB began at the prototype stage, and up to 40 app developers within the company gave their feedback to the core development team. The SDK research team at Intel also provided regular testing feedback, as did a game expert and an engineer from the Intel R&D team in Shanghai. TheBestSync used Intel® VTune Amplifier as an important tool to test and manage the system resource allocation of Finding BBB for optimum CPU and GPU performance. This included helping the game accurately judge when to activate any available multiple threads. Once the game had been developed beyond the prototype stage, the team invited friends from outside TheBestSync’s development department to participate in the testing.

After testing and analysis, the team found that the huge 3D scenes they had initially created, which were rendered in real time, conflicted with the resource allocation that the Intel RealSense SDK required. This caused the game to suffer a dip in performance, which manifested as a frame-rate drop on the target Intel® Core™ i5 processor hardware to 15 frames per second (fps)—significantly below the target of 30 fps. To address the problem, the team modified the code to give a more reasonable resource allocation to the CPU and GPU. The result was noticeably better performance and a stable frame rate of 30 fps on a wide variety of devices, including those with lower specs.

What’s Next for Finding BBB

Finding BBB has achieved TheBestSync’s vision of a social interactive game that harnesses all the core capabilities of Intel RealSense technology. However, the team has identified areas where the game can be improved. Despite having already made significant changes to improve the learning curve for the controls, TheBestSync believes that navigating the 3D environment can be made even easier and more accessible for the player. The goal is that anyone, of any age, can start to play the game and have a rewarding experience with a minimal amount of coaching. This aspect of user-friendliness is what the team is continuing to enhance.

Augmented reality and virtual reality are among TheBestSync’s areas of expertise, and the team believes that there is significant potential in combining Intel RealSense technology and virtual reality technologies such as Oculus Rift. By bringing the intuitive and natural control interface into the virtual reality arena, the company believes that it can create an experience that is immersive to a degree that has not yet been reached. The player will be able to use virtual reality to “feel” the game while using Intel RealSense technology to control it. Another potential application of this combination of technologies is the creation of navigable 3D-rendered videos that can be viewed and manipulated in 360 degrees.

Since it began working with Intel RealSense technology two years ago, TheBestSync has already seen the development community’s landscape change considerably. When it entered the Intel Perceptual Computing Challenge with JOY, only a few developers were aware of Intel RealSense technology. Since then, the development community’s engagement with this technology has mushroomed. Interest in it has even spread to a number of universities that want to learn more about the technology’s capabilities and potential. As a result of this interest and the company’s own first-hand experience with the technology, there is a great deal of excitement at TheBestSync about the future potential of Intel RealSense technology.

About the Developer

Headed by CEO Alpha Lam, TheBestSync was founded in Guangzhou, China, in 2011. From the beginning, the company’s focus has been on developing products that exploit augmented reality and perceptual computing technologies to deliver innovative user experiences. Its products were initially designed for iOS* and Android* mobile platforms, later moving to Windows* platforms. Today, the company is investing heavily in developing interactive hardware and software that make innovative use of Intel RealSense technology, such as the RealSense “Fun Cap” claw machine, which was demoed during the 2014 Intel Developer Forum keynote. Its products are being designed for the coming wave of Intel RealSense technology-based devices, which will include PCs, laptops, tablets, and the Windows Phone*.

TheBestSync has also established relationships with a number of OEMs, including Lenovo, Haier, Acer, HP, and Dell, with the goal of having its apps preloaded onto Windows* devices. The company previously created an augmented reality-based app that was bundled with the Lenovo X1 Carbon* at launch. BBB GOAL, its more recent app, is among those that will be preloaded onto forthcoming Intel RealSense technology-equipped devices, something that the company hopes to repeat with Finding BBB.

Always looking to innovate, TheBestSync is currently advancing its use of social networks such as Facebook and WeChat (in China) as app payment platforms, while also investigating the role of the Internet of Things in the experiences that it is developing. The company plans to launch Finding BBB in the spring of 2015 for Windows 8 devices with the integrated Intel RealSense 3D camera. TheBestSync will take a two-fold approach in marketing the game, working with OEMs to have the app preinstalled on Intel RealSense technology-based devices and marketing the app itself in the Windows Store.

Helpful Resources