Archived - Intel® RealSense™ SDK - Your Face as a Game Controller using JavaScript*

The Intel® RealSense™ SDK has been discontinued. No ongoing support or updates will be available.

Note: JavaScript* has been decoupled specifically for Intel® RealSense™ SDK R3. To enable JavaScript* support, please follow the tutorial.

One of the things I've been most excited about with the Intel® RealSense™ SDK is your ability to create JavaScript application.  It's one of the first things I tried and I was happy to see that it's pretty darn easy to do and it's super responsive.  Below is my approach to creating arcade style game, without having to touch, mouse or keyboard to play. Instead you move your head to play the game. Nothing super altruistic here, however as a game enthusiast think of opening up 40 years of amazing arcade video games to those who are disabled or otherwise can't use their hands to control a game.  That's just cool

I decided to start with some sample HTML5 code I wrote, (read my original post.) which shows how you can simplify controls of both aim and fire of a game player into a single tap. That post will give you and idea of what I'm converting to RealSense. However this video below shows where I ended up. This took me all about 3 hours to code while watching the 49ers lose miserably to the Oakland Raiders. So my attention was a bit split but I got the code done in the time it tool to watch the game.

How I did this:

Requirements: This is cutting edge development so to do this you'll need some recent technology such as: a Windows PC, with 4th Generation Intel Core processor and a RealSense 3D depth camera (My system is a Lenovo Yoga 3 Pro running a Core M processor). 

The foundation of this code is from the javascript Headtracking example that is part of the samples. After installing the SDK, this sample can be found at $(RSSDK_DIR)/framework/JavaScript/face_tracking.html .  This example displays all 77 points of the face that are tracked either from the IR feed or the RGB feed.   In my code sample I hard code to the RGB because that feed is higher resolution and set my game space to better fill the screen.

First step I took was to isolate the tracking to one of the 77 points.  I did this by eliminated the For loop that goes through all 77 points, then focusing on the tip of the nose only.  That allowed me one point to move around.

Source Intel® RealSense™ SDK Documentation software.intel.com
source: software.intel.com

 

Next I needed to figure out how to mirror the X position because when I moved my head right the position moved left and visa versa.  I realized that is as simple as turning the number negative and then adding half the width of the canvas ie noseX = -point.image.x+ canvas.width;.  The code below shows where the 

  if (face.landmarks.landmarksPoints !== 'undefined') {
            //            for (var i = 0; i < face.landmarks.landmarksPoints.length; i++) {
                            point = face.landmarks.landmarksPoints[29];
                            if (point != null) {
                                context.beginPath();
                                noseX = -1*point.image.x+ canvas.width;
                                 context.arc(noseX, point.image.y, 2, 0, 2 * Math.PI);
                                context.lineWidth = 2;
                                context.strokeStyle = 'red';
                                context.stroke();
                            }
             //           }
}

Now I can move and the dot follows my nose.  However the issue now is the movement is contained to about 10% of the screen.  I tested moving my neck comfortably left, right, up down to something that felt natural and it seems that my in 1080 pixels I moving about 100 pixels left to right.  So I wrote an formula to multiply the movements to fill the space based on the percent I was able to comfortably move. Thus if my screen is 1080 wide and my natural movement is 20% of that, then I multiple the Xpos by 100/20 (ie 5). And because the origin is the center we subtract then add in the center of the screen  

NewX = (noseX-halfwidth)*(100/PercentMovementX)+halfwidth
NewY = (noseY-halfheight)*(100/PercentMovementY)+halfheight

Here's how that ended up in code

if (face.landmarks.landmarksPoints !== 'undefined') {

            //            for (var i = 0; i < face.landmarks.landmarksPoints.length; i++) {
                            point = face.landmarks.landmarksPoints[21];
                            if (point != null) {
                                context.beginPath();
                                noseX = -1*point.image.x+ canvas.width;
                                noseY = point.image.y;
                                NewX = (noseX-halfWidth)*(100/CursorRatioX)+halfWidth;
                                NewY = (noseY-halfHeight)*(100/CursorRatioY)+halfHeight;
                                context.arc(NewX , NewY, 2, 0, 2 * Math.PI);
                                context.lineWidth = 2;
                                context.strokeStyle = 'red';
                                context.stroke();
                            }
             //           }
}

With this, I can now move the cursor to any point of the screen very comfortably.  It's very response and actually less tiresome than holding my hand up to control with gestures.

The next phase is to integrate my existing HTML5 code for rotating my spaceship and firing laser beams.  I grabbed the code from jfiddle here .  I quickly noticed a problem with cutting and pasting this code in.  The sample code provided is written in jquery and I had never mixed jquery and javascript functions.  So a quick look-up on the web solved the problem. I needed to make my function calls with my functions listed inside 
$(document).ready(function() {
....

}

Also I discovered the Canvas and Context variables need to be re-declared in each Function that is using Canvas Element.  With that done it was very simple

First I changed out the Canvas item that drew the circle where my nose controller and replaced with my CrossHairs function. With that done I now have a nice cross hairs graphic moving around instead of that simple circle.   I also removed the mouse event listener and set the default value of "shot" to true and remarked out laserstatus.  Those 2 little changes in code made the the game play very functional. The ship auto rotates around to pointing at my cursor, wherever my nose moves it. Thanks to the Arctangent work I did in my previous post this is a breeze.

Finally I had to figure out how to fire the laser.   If this was truly going to allow you to be hands free, I needed to find another way to trigger lasers using facial landmarks.  An eyebrow lift was the answer.  To calculate and eyebrow lift I looked at the difference between the Y position of the eyebrow and the top middle of the eye.  If that increases more than average than I triggered the laserstatus.  Below is the full code, you can try it out and do something more with it.  For me I think I will create some enemies to shoot at, and a calibration screen so my numbers for multiplying head movement and eyebrow raises are not hard coded.  Should be fun

<!DOCTYPE html>

<html>
<head>
    <title>Intel(R) RealSense(TM) SDK Web Sample</title>

    <script type="text/javascript" src="..\common\JavaScript\jquery-1.11.0.min.js"></script>
    <script type="text/javascript" src="..\common\JavaScript\promise-1.0.0.min.js"></script>
    <script type="text/javascript" src="..\common\JavaScript\realsense-3.0.js"></script>
    <script type="text/javascript" src="..\common\JavaScript\realsenseinfo-3.0.js"></script>
    <script type="text/javascript">

var noseX = 0;
var noseY = 0;
//
var canvas;
var alphaset =0;
var linealpha =alphaset.toFixed(2);
var context;
var CursorRatioX = 10;
var CursorRatioY = 10;
var gamew = 1200; //width of our game space
var gameh = 600; //height of our game space
var shipSize = 60 //size of our ship
var angleRate = .5; //reduces or increases the angle by a rate   
var xAngle = 0; //global variable for horizontal angle of the device
var yAngle = 0; //global variable for vertical angle of the device
var spin = 1;
var shipx = gamew * .5;
var shipy = gameh * .5;
var spinr;
var spinl;
var rotatespeed = 0;
var touchX = 0;
var touchY = 0;
var shot;
var xhairRadian;
var shotstart = 0;
var shipRadian;
var xhairRadianround;
var spinRound;
var speedmax = 20;
var spindefault = 0;
var spinspeed = spindefault
var direction;
var deltaRadian;
var shotprogress = false;
var laserstatus = 0;
var h = 0;
var laserstart = 0;
var right;
var xforcespeed = 0;
var yforcespeed = 0;
var left;
var down;
var up;





// setInterval(game, 25); // waits 25 milliseconds then repeats all of the above

function game() {
    canvas = document.getElementById('myCanvas');
    context = canvas.getContext('2d'); // context is the variable to envoke all Canvas commands
    context.clearRect(0, 0, gamew, gameh); // Clears the entire screen
    context.strokeStyle = "gray" // Color of the object lines
    context.lineWidth = 2;
    context.fillStyle = "grey"; // Color of the Game Space
    context.strokeRect(0, 0, gamew, gameh); //Draw Game Space  
    context.save(); // Save the canvas location

    context.strokeStyle = "white" // Color of the object lines

   
}



function crosshairs() { //draws the crosshairs
    
                var canvas = document.getElementById('myCanvas');
                var context = canvas.getContext('2d');

    context.save();
    context.translate(touchX, touchY);
    context.beginPath();
    context.moveTo(0, -20)
    context.lineTo(0, 20)
    context.moveTo(-20, 0)
    context.lineTo(20, 0)
    context.rotate(-45 * Math.PI / 180);
    context.strokeRect(-10, -10, 20, 20);
    context.restore();
    context.stroke();
}

function drawShot() { // determines if we can draw laser fire
    laserstart = 1;
    if (laserstatus > 0 && h > 0) {
        drawlaser();
        shot = false;
    }
    if (h > 7) {
        h = 0;
        laserstatus = 0;
        laserstart = 1;
    }
}

function spincalc() { //calculates the spin number to rotate the ship with variable speed

    if (shotstart == 1) { //if the shot was made start to spin the ship
        if (direction == "left") {
            spinspeed--; //if not at top speed then increase the speed of the ship turning in the negative direction
            if (spinspeed < (speedmax * -1)) {
                spinspeed = (speedmax * -1); //if you hit top speed don't increase the speed anymore
            }
        } else {
            spinspeed++; //if not at top speed then increase the speed of the ship turning in the positive direction
            if (spinspeed > speedmax) { //if you hit top speed don't increase the speed anymore
                spinspeed = speedmax;
            }
        }
        if (spin >= 360) { //if you've come all the way around, reset the spin by 360
            spin = spin - 360;
        }
        if (spin <= 0) { //if you've come all the way around, reset the spin by 360
            spin = spin + 360;
        }
        spin += spinspeed;
        spinspeed *= 1.6;
        //round out the variables so they are easier to work with
        xhairRadianround = Math.round(xhairRadian * 10) / 10;
        spinRound = Math.round(shipRadian * 10) / 10;
        spin = Math.round(spin * 100) / 100
    }
}

function drawship() {
var canvas = document.getElementById('myCanvas');
                var context = canvas.getContext('2d');
    spincalc()
    context.save(); // Save the canvas location  
    rotateship();
    makeship();
    context.restore(); // Restores our canvas (to its X&Y position - that way the game space rectangle is drawn at the same spot next time
    if (shipRadian == xhairRadian) {
        drawShot();
        shotprogress = true;
    }
}



function rotateship() { //rotates the ship
    var canvas = document.getElementById('myCanvas');
                var context = canvas.getContext('2d');
    if (spinRound >= xhairRadianround - 0.5 && spinRound <= xhairRadianround + 0.5 || spinRound > Math.PI * 2 || spinRound < 0) {; //if the ships close enough to the proper angle no need to animate just point the ship at the cursor
        shipRadian = xhairRadian;
        spinspeed = spindefault;
        shotstart = 0;
    } else { //if the angle is far enough off start to spin the ship 
        shipRadian = (spin * Math.PI / 180)
    }
    context.translate(shipx, shipy); // Move the Canvas Coordinates to the ship X & Y position (remember this moves each time)
    context.rotate(shipRadian);
}



function drawlaser() {
    var canvas = document.getElementById('myCanvas');
    var context = canvas.getContext('2d');
    context.save();
    context.beginPath();
    context.translate(shipx, shipy);
    context.rotate(shipRadian - (-90 * Math.PI / 180))
    context.moveTo(shipSize * -.3, shipSize * -.3);
    context.restore();
    context.lineTo(touchX, touchY);
    context.save();
    context.translate(shipx, shipy);
    context.rotate(shipRadian - (-90 * Math.PI / 180));
    context.moveTo(shipSize * .3, shipSize * -.3);
    context.restore();
    context.lineTo(touchX, touchY);
    alphaset = 1-(h/20);
    linealpha = alphaset.toFixed(2);
    context.strokeStyle = "rgba(255, 0, 0,"+ linealpha+")";
    context.stroke();
    context.restore();
}



function makeship() {
    var canvas = document.getElementById('myCanvas');
                var context = canvas.getContext('2d');
    context.save();
    context.rotate(-90 * Math.PI / 180);
    context.beginPath();
    context.moveTo(shipSize * -.3, shipSize * .2);
    context.lineTo(shipSize * -.4, shipSize * -.1);
    context.lineTo(shipSize * .1, shipSize * .4);
    context.lineTo(0, shipSize * .5);
    context.lineTo(shipSize * -.1, shipSize * .4);
    context.lineTo(shipSize * .4, shipSize * -.1);
    context.lineTo(shipSize * .3, shipSize * .2);
    context.moveTo(shipSize * -.4, shipSize * -.1);
    context.lineTo(shipSize * .3, shipSize * -.5);
    context.lineTo(0, shipSize * .3);
    context.lineTo(shipSize * -.3, shipSize * -.5);
    context.lineTo(shipSize * .4, shipSize * -.1);
    context.strokeStyle = "white";
    context.stroke();
    context.restore();
}
        
        
        
        
//        ---
        $(document).ready(function () {
            var sense;
            var imageSize;
            var faceConfiguration;

            // check platform compatibility
            RealSenseInfo(['face3d'], function (info) {
                if (info.IsReady == true) {
                    $('#info').append('<b>Platform supports Intel(R) RealSense(TM) SDK feature</b>');
                    status('OK');
                    document.getElementById("Start").disabled = false;
                } else {
                    status('Platform not supported: ' + info.responseText);
                    if (info.IsPlatformSupported != true) {
                        $('#info').append('<b>Intel� RealSense� 3D camera not found</b>');
                    } else if (info.IsBrowserSupported != true) {
                        $('#info').append('<b>Please update your browser to latest version</b>');
                    } else {
                        $('#info').append('<b>Please download and install the following update(s) before running sample: </b>');
                        for (i = 0; i < info.Updates.length; i++) {
                            $('#info').append('<a href="' + info.Updates[i].url + '">' + info.Updates[i].href + '</a><br>');
                        }
                    }
                }
            })

            $('#Start').click(function () {
                document.getElementById("Start").disabled = true;
                PXCMSenseManager_CreateInstance().then(function (result) {
                    sense = result;
                    return sense.EnableFace(onFaceData);
                }).then(function (result) {
                    return result.CreateActiveConfiguration();
                }).then(function (result) {
                    faceConfiguration = result;
                    faceConfiguration.configs.detection.isEnabled = document.getElementById("detection").checked;
                    faceConfiguration.configs.landmarks.isEnabled = document.getElementById("landmarks").checked;
                    faceConfiguration.configs.pose.isEnabled = document.getElementById("pose").checked;
                    faceConfiguration.configs.expressionProperties.isEnabled = document.getElementById("expressions").checked;
                    var selects = document.getElementById("mode");
                    var mode = 0;// Number(selects.options[selects.selectedIndex].value);
                    return faceConfiguration.SetTrackingMode(mode);
                }).then(function (result) {
                    return faceConfiguration.ApplyChanges();
                }).then(function (result) {
                    status('Init started');
                    return sense.Init(null, onStatus);
                }).then(function (result) {
                    return sense.QueryCaptureManager();
                }).then(function (capture) {
                    return capture.QueryImageSize(pxcmConst.PXCMCapture.STREAM_TYPE_COLOR);
                }).then(function (result) {
                    imageSize = result.size;
                    return sense.StreamFrames();
                }).then(function (result) {
                    status('Streaming ' + imageSize.width + 'x' + imageSize.height);
                    document.getElementById("Stop").disabled = false;
                }).catch(function (error) {
                    status('Init failed: ' + JSON.stringify(error));
                    document.getElementById("Start").disabled = false;
                });
            });

            function clear() {
                $('#pose_status').text('');
                $('#expressions_status').text('');
                document.getElementById("Start").disabled = false;
                var canvas = document.getElementById('myCanvas');
                var context = canvas.getContext('2d');
                context.clearRect(0, 0, canvas.width, canvas.height);
            }

            $('#Stop').click(function () {
                document.getElementById("Stop").disabled = true;
                sense.Close().then(function (result) {
                    status('Stopped');
                    clear();
                });
            });

            function onFaceData(mid, module, data) {
                var canvas = document.getElementById('myCanvas');
                var context = canvas.getContext('2d');

                var headTolerance = 10;

                canvas.width = imageSize.width;
                canvas.height = imageSize.height;
                shipx = canvas.width/2;
                shipy = canvas.height/2;

                if (data.faces == 'undefined') return;
                for (f = 0; f < data.faces.length; f++) {
                    var face = data.faces[f];
                    if (face.detection != null) {
                        if (face.detection.faceBoundingRect !== 'undefined') {
                            var rectangle = face.detection.faceBoundingRect;
                            context.beginPath();
                            context.lineWidth = 10;
                            context.strokeStyle = "rgba(0, 255, 0, 0.1)";
                            context.rect(-1*rectangle.x+canvas.width, rectangle.y, -rectangle.w, rectangle.h);
                           // context.rect(rectangle.x, rectangle.y, rectangle.w, rectangle.h);
                            context.stroke();
                        }
                    }
                    if (face.landmarks.landmarksPoints !== 'undefined') {
                       // for (var i = 0; i < face.landmarks.landmarksPoints.length; i++) {
                        var browpoint = face.landmarks.landmarksPoints[7];
                        var mideye = face.landmarks.landmarksPoints[20];
                        var raisedbrow= mideye.image.y - browpoint.image.y;
                            point = face.landmarks.landmarksPoints[29];
                            if (point != null) {
                                shot=true;
                                noseX = -1*point.image.x+ canvas.width;
                                noseY = (point.image.y );
                                touchX = (noseX-shipx)*(100/CursorRatioX)+shipx
                                touchY = (noseY-shipy)*(100/CursorRatioY)+shipy
                                console.log(noseY);
                                context.beginPath();
                                
                              //  context.arc(point.image.x, noseY, 2, 0, 2 * Math.PI);
                                context.lineWidth = 2;
                                context.strokeStyle = 'red';
                                context.stroke();
                                if (laserstatus > 0 && laserstart > 0) { // draw color the laser red when fired
                                    h++;
                                }
                                $(document).ready(function() {
                                    //crosshairs();
                                    context.fillStyle="red";
                                    context.font = "50px Arial";
                                  //  context.fillText(linealpha,200,200);
                                    context.arc(noseX, noseY, 2, 0, 2 * Math.PI);
                                    crosshairs();
                                    drawship();
                        context.save();
                                    context.restore();
                                });

    



    if (shot) { // when clicking mouse this we calculate the angle to the point clicked
       // laserstatus = 1;
        if (raisedbrow >68){
        laserstatus = 1;
        }
        else {
            laserstatus = 0;
        }
        xhairRadian = Math.atan2(touchY - shipy, touchX - shipx); // Calculate radian angle of target fire
        if (xhairRadian <= 0) { // The above calulate a negative radian. Turn the negative radian into is positive counterpart
            xhairRadian = 2 * Math.PI + xhairRadian;
        }
        deltaRadian = xhairRadian - shipRadian
        if (deltaRadian < -Math.PI || deltaRadian > Math.PI) { // determine if the spin direction should be left or right
            if (xhairRadian < shipRadian) {
                direction = "right";
            }
            if (xhairRadian > shipRadian) {
                direction = "left";
            }
        } else { // else if the difference in angle is positive spin toward the right
            if (xhairRadian > shipRadian) {
                direction = "right";
            }

            if (xhairRadian < shipRadian) { /// if the difference in angls is negative spin toward the left
                direction = "left";
            }
        }
        shotstart = 1; // shotstart = 1 means we've finished the calculations and are ready to shoot the laser at the target
    }
                            }
                        //}
                    }
                    if (face.pose !== 'undefined' && face.pose != null) {
                        $('#pose_status').text('Pose: ' + JSON.stringify(face.pose));
                    } else {
                        $('#pose_status').text('');
                    }
                    if (face.expressions !== 'undefined' && face.expressions != null) {
                        $('#expressions_status').text('Expressions: ' + JSON.stringify(face.expressions));
                    } else {
                        $('#expressions_status').text('');
                    }
                }
            }

            function onStatus(data) {
                if (data.sts < 0) {
                    status('Error ' + data.sts + ' on module ' + data.mid);
                    clear();
                }
            }

            function status(msg) {
                $('#status').text(msg);
            }
        });
    </script>
</head>

<body bgcolor="black">
    <div id="wrapper">
        <div id="container">

                <div id="info"></div>
            <br>
          

            <canvas id="myCanvas" style="border:1px solid #000000;  background-color:black"></canvas>

            <div id="pose_status"></div>
            <div id="expressions_status"></div>
              <select id="mode">
                <option value=0>2D</option>
                <option value=1 selected="selected">3D</option>
            </select>
            Mode
            <input id="detection" type="checkbox" checked="checked"" />Detection
            <input id="landmarks" type="checkbox" checked="checked"" />Landmarks
            <input id="pose" type="checkbox"" />Pose
            <input id="expressions" type="checkbox"" />Expressions
            <br>
            <br>
            <button id="Start">Start</button>
            <button id="Stop" disabled="disabled">Stop</button>
            <br>
            <br>
            <br>
            Status:<div id="status"></div>
        </div>
    </div>
</body>
</html>

 

有关编译器优化的更完整信息,请参阅优化通知