Infrared5 Ultimate Coder Week Six: GDC, Mouth Detection Setbacks, Foot Tracking and Optimizations Galore

For week six Chris and Aaron made the trek out to San Fransisco to the annual Game Developers Conference (GDC) where they showed the latest version of our game Kiwi Catapult Revenge. The feedback we got was amazing! People were blown away at the head tracking performance that we’ve achieved, and everyone absolutely loved our unique art style. While the controls were a little difficult for some, that allowed us to gain some much needed insight into how to best fine tune the face tracking and the smartphone accelerometer inputs to make a truly killer experience. There’s nothing like live playtesting on your product!



Not only did we get a chance for the GDC audience to experience our game, we also got to meet some of the judges and the other Ultimate Coder competitors. There was an incredible amount of mutual respect and collaboration among the teams. The ideas were flowing on how to help improve each and every project in the competition. Chris gave some tips on video streaming protocols to Lee so that he will be able to stream over the internet with some decent quality (using compressed JPEGs would have only wasted valuable time). The guys from Sixense looked into Brass Monkey and how they can leverage that in their future games, and we gave some feedback to the Code Monkeys on how to knock out the background using the depth camera to prevent extra noise that messes with the controls they are implementing. Yes, this is a competition, but the overall feeling was one of wanting to see every team produce their very best.


The judges also had their fair share of positive feedback and enthusiasm. The quality of the projects obviously had impressed them, to the point that Nicole was quoted saying “I don’t know how we are going to decide”. We certainly don’t envy their difficult choice, but we don’t plan on making it any easier for them either. All the teams are taking it further and want to add even more amazing features to their applications before the April 12th deadline.


The staff in the Intel booth were super accommodating, and the exposure we got by being there was invaluable to our business. This is a perfect example of a win-win situation. Intel is getting some incredible demos of their new technology, and the teams are getting exposure and credibility by being in a top technology company’s booth. Not only that, but developers now get to see this technology in action, and can more easily discover more ways to leverage the code and techniques we’ve pioneered. Thank you Intel for being innovative and taking a chance on doing these very unique and experimental contests!


While Aaron and Chris were having a great time at GDC the rest of the team was cranking away. Steff ran into some walls with mouth detection for the breathing fire controls, but John, Rebecca and Elena were able to add more polish to the characters, environment and game play.



John added on a really compelling new feature - playing the game with your feet! We switched the detection algorithm so that it tracks your feet instead of your face. We call it Foot Tracking. It works surprisingly well, and the controls are way easier this way.



Steff worked on optimizing the face tracking algorithms and came up with some interesting techniques to get the job done.


This week’s tech tip and code snippet came to us during integration. We were working hard to combine the head tracking with the Unity game on the Ultrabook, and ZANG we had it working! But, there was a problem. It was slow. It was so slow it was almost unplayable. It was so slow that it definitely wasn’t “fun.” We had about 5 hours until Chris was supposed to go to the airport and we knew that the head tracking algorithms and the camera stream were slowing us down. Did we panic? (Don’t Panic!) No. And you shouldn’t either when faced with any input that is crushing the performance of your application. We simply found a clever way to lower the sampling rate but still have smooth output between frames.


The first step was to reduce the number of times we do a head tracking calculation per second. Our initial (optimistic) attempts were to update in realtime on every frame in Unity. Some computers could handle it, but most could not. Our Lenovo Yoga really bogged down with this. So, we introduced a framesToSkip constant and started sampling on every other frame. Then we hit a smoothing wall. Since the head controls affect every single pixel in the game window (by changing the camera projection matrix based on the head position), we needed to be smoothing the head position on every frame regardless of how often we updated the position from the camera data. Our solution was to sample the data at whatever frame rate we needed to preserve performance, save the head position at that instant as a target, and ease the current position to the new position on every single frame. That way, your sampling rate is down, but you’re still smoothing on every frame and the user feels like the game is reacting to their every movement in a non-jarring way. (For those wondering what smoothing algorithm we selected:  Exponential Smoothing handles any bumps in the data between frames.) Code is below.

using UnityEngine;
using System.Collections;
using System;
using System.Runtime.InteropServices;

public class IntelPerCompController : MonoBehaviour 
{
 private Vector3 facePosition = new Vector3(0.0f, 0.0f, 0.5f);
 private Vector3 targetFacePosition = new Vector3(0.0f, 0.0f, 0.5f);
 private FaceTrackerWrapper faceTracker;
 private bool faceTrackingIsWorking = false;
 // storage for the number of frames we have to play until we need to get an update from
 // the camera (on head position)
 private uint cntToNextUpdate = 0;
 // the number of frames to skip to reduce the sampling rate to the camera
 private const uint framesToSkip = 2;
 private bool showOpenCVWindow = false;
 private bool gotUpdateFromCamera = false;

 public Vector3 fireDirection;

 void Start() 
 {
 Debug.Log("IntelPerCompController :: Start");
 fireDirection = new Vector3(0, 0, 1);
 faceTracker = new FaceTrackerWrapper();
 // location of the haar cascade file for openCV to load in the DLL
 string haarPath = @"./Assets/haarcascade_frontalface_alt.xml";
 int a = faceTracker.InitTracking(haarPath);
 // save if we initialized successfully
 faceTrackingIsWorking = (a == 0);
 // output result to log
 Debug.Log("faceTrackingIsWorking = " + faceTrackingIsWorking);
 }


 void OnDestroy()
    {
 //Debug.Log("OnDestroy");
 // shut down the camera
 //faceTracker.EndTracking();
    }


 void Update()
 {
 if (faceTrackingIsWorking)
 {
 gotUpdateFromCamera = false;
 // check if we're due to update the head position from the camera data
 if (cntToNextUpdate == framesToSkip)
 {
 // always check if we have successfully advanced frames
 gotUpdateFromCamera = faceTracker.AdvanceFrame();
 //Debug.Log("gotNewFrame = " + gotNewFrame);

 if (gotUpdateFromCamera)
 {
 // get the values for the new face position
 float newX = faceTracker.GetFaceX();
 float newY = faceTracker.GetFaceY();
 float newZ = faceTracker.GetFaceZ();
 Vector3 newPos = new Vector3(newX, newY, newZ);
 // save the new value in facepos for the projection calc and to smooth on the next frame
 targetFacePosition.x = newPos.x;
 targetFacePosition.y = newPos.y;
 targetFacePosition.z = newPos.z;
 //Debug.Log("facePosition = " + facePosition);

 // reset the count to the next camera update
 cntToNextUpdate = 0;
 }
 }
 else
 {
 // count until we need to make a new update
 cntToNextUpdate++;
 }
 // smooth on every frame toward the last target position from the camera
 facePosition = DataSmoothingUtil.ExponentialSmoothing3(targetFacePosition, facePosition, 0.18f);
 }
 // check if the user is holding down the left cntrl key
 if (Input.GetKey(KeyCode.LeftControl))
 {
 // if w, toggle the results window
 if (Input.GetKeyDown(KeyCode.W))
 {
 showOpenCVWindow = !showOpenCVWindow;
 faceTracker.ShowResultsWindow(showOpenCVWindow);
 }
 // if q, quit the app
 if (Input.GetKeyDown(KeyCode.Q))
 {
 Application.Quit();
 }
 }
 }


 void LateUpdate() 
 {
 // still update every frame to show the last smoothed result
 if (faceTrackingIsWorking)
 {
 float n = Camera.main.nearClipPlane;
 float f = Camera.main.farClipPlane;

 // all below in real world space
 // screen's bottom left corner 
 Vector3 pa = new Vector3(-0.145f, -0.135f, 0.02f);
 // screen's bottom right corner
 Vector3 pb = new Vector3(0.145f, -0.135f, 0.02f);
 // screen's top left corner
 Vector3 pc = new Vector3(-0.145f, 0.1f, 0.0f);
 // face position 
 Vector3 pe = new Vector3(-facePosition.x, -facePosition.y, facePosition.z);
 //Debug.Log("pe: " + pe);

 Camera.main.projectionMatrix = generalizedPerspectiveProjection(pa, pb, pc, pe, n, f);

 // calculate the direction that things should fire (so it feels like it is coming from your head)
 Vector3 p = Camera.main.ViewportToWorldPoint(new Vector3(0.5F, 0.5F, 10.0F));
 //q.SetFromToRotation(p, Camera.main.transform.position);
 // store the fireDirection as a normalized vector (for fun!! or possible smoothing later if needed)
 p = p - Camera.main.transform.position;
 fireDirection = p.normalized;
 Quaternion q = new Quaternion();
 q.SetLookRotation(fireDirection);
 PlayerController.Instance.CurrentHeadRotation = q;
 }
 }


 Matrix4x4 generalizedPerspectiveProjection(Vector3 pa, Vector3 pb, Vector3 pc, Vector3 pe, float n, float f) 
 {
 // Compute an orthonormal basis for the screen.
 Vector3 vr = pb - pa;
 vr.Normalize();
 Vector3 vu = pc - pa;
 vu.Normalize();
 Vector3 vn = Vector3.Cross(vr, vu);
 vn.Normalize();

 // Compute the screen corner vectors.
 Vector3 va = pa - pe;
 Vector3 vb = pb - pe;
 Vector3 vc = pc - pe;

 // Find the distance from the eye to screen plane.
 float d = -Vector3.Dot(va, vn);

 // Find the extent of the perpendicular projection.
 float m = n / d;
 float l = Vector3.Dot(vr, va) * m;
 float r = Vector3.Dot(vr, vb) * m;
 float b = Vector3.Dot(vu, va) * m;
 float t = Vector3.Dot(vu, vc) * m;

 // projection matrix 
 Matrix4x4 p = Matrix4x4.identity;
 p[0,0] = 2.0f * n / (r - l);
 p[0,1] = 0.0f;
 p[0,2] = (r + l)/(r - l);
 p[0,3] = 0.0f;

 p[1,0] = 0.0f;
 p[1,1] = 2.0f * n / (t - b);
 p[1,2] = (t + b) / (t - b);
 p[1,3] = 0.0f;

 p[2,0] = 0.0f;
 p[2,1] = 0.0f;
 p[2,2] = (f + n) / (n - f);
 p[2,3] = 2.0f * f * n / (n - f);

 p[3,0] = 0.0f;
 p[3,1] = 0.0f;
 p[3,2] = -1.0f;
 p[3,3] = 0.0f;

 // rotation matrix;
 Matrix4x4 rm = Matrix4x4.identity;
 rm[0,0] = vr.x;
 rm[0,1] = vr.y;
 rm[0,2] = vr.z;
 rm[0,3] = 0.0f;

 rm[1,0] = vu.x;
 rm[1,1] = vu.y;
 rm[1,2] = vu.z;
 rm[1,3] = 0.0f;

 rm[2,0] = vn.x;
 rm[2,1] = vn.y;
 rm[2,2] = vn.z;
 rm[2,3] = 0.0f;

 rm[3,0] = 0.0f;
 rm[3,1] = 0.0f;
 rm[3,2] = 0.0f;
 rm[3,3] = 1.0f;

 // translation matrix;
 Matrix4x4 tm = Matrix4x4.identity;
 tm[0,0] = 1.0f;
 tm[0,1] = 0.0f;
 tm[0,2] = 0.0f;
 tm[0,3] = -pe.x;

 tm[1,0] = 0.0f;
 tm[1,1] = 1.0f;
 tm[1,2] = 0.0f;
 tm[1,3] = -pe.y;

 tm[2,0] = 0.0f;
 tm[2,1] = 0.0f;
 tm[2,2] = 1.0f;
 tm[2,3] = -pe.z;

 tm[3,0] = 0.0f;
 tm[3,1] = 0.0f;
 tm[3,2] = 0.0f;
 tm[3,3] = 1.0f;

 return p * rm * tm;
    }
}

Feeling good about the result, we went after mouth open/closed detection with a vengeance! We thought we could deviate from our original plan of using AAM and POSIT, and lock onto the mouth using a mouth specific Haarcascade on the region of interest containing the face. The mouth Haarcascade does a great job finding and locking onto the mouth if the user is smiling - which is not so good for our purposes. We are still battling with getting a good lock on the mouth using a method that combines depth data with RGB, but we have seen why AAM exists for feature tracking. It’s not just something you can cobble together and have confidence that it will work well enough to act as an input for game controls.


Overall, this week was a step forward even with part of the team away. We’ve got some interesting and fun new features that we want to add as well. We will be sure to save that surprise for next week. Until then, please let us know if you have any questions and/or comments. May the best team win!

Per informazioni complete sulle ottimizzazioni del compilatore, consultare l'Avviso sull'ottimizzazione