Aircraft Smooth Motion Controls with Intel® Perceptual Computing SDK

Download Article


Aircraft Smooth Motion Controls with Intel® Perceptual Computing SDK [PDF 1.3MB]

1 Introduction


This document explains how to use the data provided by the Intel® Perceptual Computing SDK to implement smooth motion controls in a 3D game. The algorithm explained in this paper was used to create a demo with Unity* and the Intel Perceptual Computing SDK where the player controls a plane in a 3D world.

To understand how to achieve smooth motion controls, we will first explain how the aircraft moves in a 3D world. Then we will take a look at the data produced by the SDK and show our first version of the motion controls and an algorithm for it. In the last part, we will explain how to implement this algorithm in Unity. This paper focuses on the methodology we used to get the proposed solution.

2 Playing with the aircraft orientation


2.1 The forces in our game

Our goal is not to create a realistic flight simulator, we just want the plane to move in a way that the player thinks is normal. We also want to minimize the impact of the motion controls on the CPU workload because in a game, the CPU usually has a lot of other jobs to process. A good solution to simplify this problem is to use the direction vector instead of the forces vectors. Figure 1 shows what happens when the plane is flying straight forward, up, and down.



Figure 1: Flying in three different directions

As we can see from the figure, the plane orientation is exactly the same as the direction vector. Taking only this vector in to consideration simplifies the problem. This representation is very close to what happens in the real world as long as the speed of the plane is fast enough. If the speed is not fast enough, the plane will fall, which is called a stall. Figure 2 shows an example of stall. Implementing this behavior will not be explained in this document and to simplify the problem, our plane will always have a speed sufficient to avoid stalls.



Figure 2: Representation of a stall

2.2 Plane orientation and direction vector

In a game, your plane’s position must be refreshed at each frame. Fortunately, as we’ve noticed in the previous section, the plane orientation and the direction vectors are the same. In most of the 3D engines, you can easily retrieve the orientation of your objects and their positions as well. With those two values, and with the speed of your plane, you can compute its position at the next frame. To simplify the explanations, the unit of the speed will be a distance by frame. The formula used to compute the plane’s position at the current frame is:

position = lastPosition + planeOrientation * speed

position: A 3D vector that holds the position of the plane at the current frame.
lastPosition: A 3D vector that holds the position at the previous frame.
planeOrientation: A 3D vector that holds the orientation of the plane at the current frame.
speed: A float that holds the speed (distance / frame).

The equation would look similar to:


 

This calculus will have to be done at each frame. We will soon see the only parameter we are going to play with is the plane orientation. This will allow the plane to move in an almost realistic way in the 3D world.

3 Working with the SDK data


3.1 Retrieving the data

You can use C++ and C# to access the Intel Perceptual Computing SDK data. Retrieving the data can be done in different ways. You can inherit the UtilPipeline class and read the data from the method:

bool OnNewFrame( void )

Or you can request a frame with the method:

bool AcquireFrame( bool wait )

The wait parameter can be set to true to wait until a new frame is ready. This also creates a lock on the frame so the function

bool ReleaseFrame( void )

will have to be called to release the lock.

In a game, we want to evaluate the controls at each frame. For this reason, we will retrieve the SDK data with the second method. Here is a snippet of code in C# (Unity):

	i f	(! pp. AcquireFrame( true ) ) return ;
	/*
	* Put some code here to	access SDK data
	*/
	pp. ReleaseFrame () ;
	/*
	* Process the data here to	create smooth controls .
	*/

Data processing will be done outside of the critical section to avoid any unnecessary lock.

Once the frame is locked, we can request an access to the data by calling the function:

bool QueryGeoNode(PXCMGesture.GeoNode. Label body ,
out PXCMGesture.GeoNode data

This function takes two parameters:
body: Describes the node(s) you are interested in.
data: Contains the position of the selected nodes.

This function also returns false if the requested nodes are not available, which occurs if the player’s hands are not recognizable by the SDK. If the function returns true, the node positions can be read in the data parameter. In our case, we are requesting information between the two hands.

When specifying the body parameter, you must specify the hand and the node(s) of interest. To read data from the left hand, you need to send:


	PXCMGesture.GeoNode. Label .LABEL BODY HAND LEFT |

	PXCMGesture.GeoNode. Label .LABEL HAND MIDDLE

	

3.2 Position world

In the previous part, we explained how to retrieve hand positions with a call to:


	bool QueryGeoNode(PXCMGesture.GeoNode. Label body ,

	out PXCMGesture.GeoNode data

	

Now that the variable data is set, we want to retrieve a 3D vector with the position of the hand in 3D space. This can be achieved with the positionWorld attribute. You must keep in mind that this attribute represents the position of player’s hands from the camera point of view (not the screen!). positionWorld has three attributes (x, y, and z) that you can use to get the location of player’s hands.

Figure 3 shows the axis representation used in the SDK.



Figure 3: Axis representation

4 Motion controls implementation


4.1 The gameplay

Now that the data can be accessed, it’s time to define the gameplay, or the movements players make to go up, down, turn, etc. We explained previously that the plane will have a fixed speed so there is no need of any control on this side. The following section describes the plane motion controls associated with the player movements.

4.1.1 Changing the pitch

The pitch can be described as the orientation of the plane’s nose regarding the horizon. For this part, we want to use a movement natural to someone who has piloted a real plane. When flying a plane, to orient the nose in the direction of the sky (go up), you pull back on the joystick. If you want to orient the nose in the direction of the ground (go down), you push the joystick forward.

In the game, we will implement the following behavior (see Figure 4):

To orient the nose in the direction of the ground, players will bring their hands close to the camera. If they want to orient the nose in the direction of the sky, they will move their hands away from the camera.



Figure 4: Controlling the pitch

4.1.2 Changing the roll

Rolling is the ability to change the angle of the wings regarding the horizon. When a plane rolls left (resp. right), its left (resp. right) wing is under the horizon and the right (resp. left) one is on the horizon.

To map the rolling, we want to use a natural movement. Our choice is as follows:

To roll left, players must put their left hands down and right hands up. To roll right, players must put their right hands down and left hands up.

That way, the plane will follow the hand movements as shown in Figure 5.



Figure 5: Controlling the roll

4.1.3 Changing the yaw

The yaw can be described as the possibility to turn without rolling. It is a rotation around the vertical axis. Concerning the game play (Figure 6), we choose to use the following movements:

To yaw left, pull your left hand. To yaw right, pull your right hand.



Figure 6: Controlling the yaw

4.2 Camera calibration

When implementing the motion controls, you will need to define a reference point. This point will be used to determine the amplitude of the movements. For example, if your two hands are close to the reference point, your plane should go straight forward. If you pull your hands just a little bit, your plane should rotate in direction of the sky very slowly. On the other hand, if you pull hard, your plane should rotate really fast.

When players start to play, they must be able to define a reference point in 3D space from which the commands will be computed. Developers should not choose the reference point because their apps could be used with a lot of different cameras and an absolute value would not work in all cases.

The best solution is to let the players choose (when they want) this point. Figure 7 illustrates the reference point.



Figure 7: Reference point and pitch

In our game, players can define the reference point by putting their thumbs up. As long as their thumbs are up, motion controls are not activated. This way, players can redefine the reference point whenever they want.

4.3 How to define the model’s rotation

Now that the game play is defined, we must link it with how rotations will be applied to the plane.

4.3.1 Implementation of the pitch

Implementing the pitch is not very complicated once you have your reference point set. To change the pitch, we just need to rotate the model around its x axis. For now, the only missing value is the rotation angle along this axis. This angle is a function that depends on the distance between the hands and the reference point. The problem is that the player’s two hands will never be at the exact same Y distance from the reference point. Figure 8 illustrates this problem.



Figure 8: Y distance from the reference point

In our implementation, the solution is quite simple. To compute the pitch angle, we always use the y value of the closest hand to the camera. In Figure 8, we would have chosen the left hand (because y1 < y2).

Right now, let’s say our game has 30 frames per seconds. The attribute positionWorld (that we are using) returns the hand position (from the camera’s point of view) in meters. So when users are playing, the values will probably be between -0.4 and 0.4 meters if they want to rotate very fast. With a game around 30 fps we get :

30 × 0.4 = 12 degrees per second

It’s definitely not a big rotation speed value, but if we just add a coefficient in the multiplication, we can rotate very quickly. Here is how to implement this function in generic language:

/*
* mainHandY: The Y distance between the main hand 
*            and the camera
* secondaryHandY: The Y distance between the secondary 
*                 hand and the camera
*/
void controlPitch(float mainHandY, float secondaryHandY){
	//check which hand is the closest to the camera
	float positionY = (mainHandY<secondaryHandY) ? 
mainHandY : secondaryHandY;
	//calibrationY is the Y position of the reference point
	float pitch = calibrationY - positionY;
	//sensibilityFactor is a coefficient used to increase
	//the rotation speed
	pitch *= sensibilityFactor;
	transform.RotateAroundLocal(transform.right, pitch);
}

4.3.2 Implementation of the roll

The roll is a rotation along the Y axis. Roll can be done by moving one hand up to the same height as the other (also see Section 4.1.2). This time, the reference point will not be used because the only thing that matters is the height difference between the left and the right hands (Figure 9).



Figure 9: Z distance between the hands

The rotation is simple to compute:

roll = mainHandZ − secondaryHandZ

For the orientation of your plane, world coordinates, and implementation, you may have to reverse the rotation. To do that, just apply a -1 factor to the roll or reverse the subtraction parameters:

roll = secondaryHandZ − mainHandZ

Once again, you can multiply your roll by a coefficient to increase the rotation speed. See the implementation in generic language:

/*
 * mainHandZ: The Z position of the main hand in 
 *            world coordinates
 * secondaryHandZ: The Z position of the secondary  
 *                 hand in world coordinates
 */
void controlRoll(float mainHandZ, float secondaryHandZ){
	float roll = mainHandZ - secondaryHandZ;
	//sensibilityFactor is a coefficient that  
	//we use to increase the rotation speed
	roll *= sensibilityFactor;
	RotateAroundYAxis(myPlane, roll);
}

As you can see, the two code snippets are symmetric.

4.3.3 Implementation of the yaw

The yaw works in the same way as the roll but with a rotation around the Z axis. This time, we need to measure the distance between the hands along the Y axis. Figure 10 illustrates this point.



Figure 10: Y distance between the hands

The rotation is computed as follows:
                          yaw = mainHandY − secondaryHandY

If the motion controls are reversed, you just need to reverse the subtraction:
                       yaw = secondaryHandY − mainHandY

Here is the code in a generic language:


	/*

	* mainHandY: The Y distance between the main hand

	*            and the camera

	* secondaryHandY: The Y distance between the 

	*                 secondary hand and the camera

	*/

	void controlYaw(float mainHandY, float secondaryHandY){

	float yaw = mainHandY - secondaryHandY;

	//sensibilityFactor is a coefficient that

	//we use to increase the rotation speed

	yaw *= sensibilityFactor;

	RotateAroundZAxis(myPlane, yaw);

	}

	

4.3.4 Section Conclusion

Implementing the motion control this way was not too hard, but the result is pretty bad. Whether you are close or far from the reference point, rotation acceleration regarding your hand positions is linear. What we really want is to have very small moves when the hands are close to the reference point, and large moves when the hands are far from this point. We’re going to show you how to do this in the next section.

5 Smoothing the motion controls


5.0.5 Analyzing the data

If you take a look at the data we are manipulating, the pitch, roll, and yaw computed at each frame are between -0.06 and 0.06. Figure 11 shows the distribution of the motions realized by the player around the reference point. As we can see, most of the moves of the player are very small. This means that most of the time, the player is looking for accuracy. What we want is to decrease the rotation factor of small moves and to increase the rotation factor of larger moves. This way, it would be possible to have very accurate controls when you are close to the reference point and very fast rotations when you are far from it.



Figure 11: Pitch distribution on a flight

5.0.6 A first solution

In an actual game, motion controls should not be a CPU consuming task. Indeed, the CPU should be used to compute physics, animate characters, etc. If smooth motion controls means a lot of CPU work, the rest of the game’s performance will suffer.

The well-known square function, which can decrease small values and increase bigger ones, seems to be well adapted to our problem. Let’s try it with our data (we will only use absolute values here). Figure 12 shows the square function applied to the pitch values.



Figure 12: The square function

We are not too far from what we would like to get. We can see in Figure 12 that small values are even smaller. This would allow players to have really accurate motion controls when they are close to the reference point. But the problem here is that we still can’t get a high rotation factor.

5.0.7 Our solution

This problem can be solved easily. The square function has a very interesting property.

We first need to decide the gap from which the rotation will increase very fast. Let’s say that we would like to increase the rotation speed when the distance is greater or equal to 0.2 meters. This means that when we reach a distance greater than 0.2 meters, we must send a value greater than 1 to the square function. Actually, we just need to remap the values from 0 to 0.2 to between 0 and 1. A simple multiplication can do the job.

1/0.2 = 5

We just need to multiply our values by 5 before sending them to the square function.

But we are not completely done because what we really want is a function that decreases the values smaller than x=0.2 and increases the values greater than x=0.2.

f(x) = (x * 5) * (x * 5)



Figure 13: The f(x)=(x*5)*(x*5) function

Figure 13 shows the new square function. As you can see, we are far from what we want.

Something is missing, and we can guess that we are looking for a function close to:

Dividing by z should allow us to get what we are looking for. Here is how to find the value of z.

What we know is: f(0.2) = 0.2

Then:

Figure 14 shows the graph of this function. We can see that it is exactly what we are looking for.

The new curve crosses the f(x)=x curve for x=0.2.



Figure 14: The f(x)=x*x*5 function

5.0.8 Section Conclusion

The result is important, but it’s more important to understand how to get there. In this part, we have seen how to smooth the first version of motion controls with the Intel Perceptual Computing SDK. As you can see in this case, we applied a modified square function to the values returned by the SDK. In addition to smooth motion controls, such a function is very easy to compute and is definitely not a CPU consuming task.

6 Integration in Unity


6.1 Introduction

In this part, we will explain how to implement this specific algorithm to control an object (a plane in this example) in Unity. This paper will not explain Unity’s basics. We will just focus on the algorithm implementation with the SDK.

6.2 Project settings

In Unity, the Intel Perceptual Computing SDK works as a plugin. The bad news is that plugin utilization is limited to Unity Pro. If you don’t have Unity Pro, but want to give it a try and experiment with the SDK, you can get a free trial of Unity Pro for 30 days.

The first thing to do is to create a folder named Plugins in your Assets directory.

The Intel Perceptual Computing SDK plugin must be added in the directory: <project>/Assets/Plugins. Those files can be retrieved in:

<SDKInstallationFolder>/framework/Unity/hellounity/Assets/Plugins. You can copy all the files.

6.3 PlaneMotion.cs script

Now we need to create a C# script that we name PlaneMotion.

This script must be attached to our plane.

The main part of the script is contained in the Update() function.

public class PlaneMotion : MonoBehaviour
{
    //We must use the gesture mode (we will use gesture 
    //matching to calibrate the camera)
    private PXCUPipeline.Mode mode = PXCUPipeline.Mode.GESTURE;
    //The UtilPipeline wrapper
    private PXCUPipeline pp;
    //We also define a speed. This parameter is public  
    //and can be redefined easily in Unity.
    public float speed = 2.0f;
    //calibrated indicates whether or not the game 
    //is calibrated
    public bool calibrated;
    //calibrationY holds the y position of the 
    //reference point.This is the only value that 
    //matters.
    private float calibrationY;
    void Start()
    {
        pp = new PXCUPipeline();
        pp.Init(mode);
        calibrated = false;
    }

    void Update()
    {
        PXCMGesture.GeoNode mainHand;
        PXCMGesture.GeoNode secondaryHand;

        //Compute the rotation with the hand position
        if (!pp.AcquireFrame(true)) return;

        if (pp.QueryGeoNode(
  PXCMGesture.GeoNode.Label.LABEL_BODY_HAND_LEFT | 
            PXCMGesture.GeoNode.Label.LABEL_HAND_MIDDLE, 
  out mainHand) 
            &&
 	  pp.QueryGeoNode(
  PXCMGesture.GeoNode.Label.LABEL_BODY_HAND_RIGHT | 
  PXCMGesture.GeoNode.Label.LABEL_HAND_MIDDLE, 
 out secondaryHand))
        {
            //We shouldn't need the following line but  
            //I already encountered some troubles.
            //This function will be explained later.
            checkHands(ref mainHand, ref secondaryHand);

            if (!calibrated)
            {
                //If the reference point is not set
                //we try to calibrate
                calibrate(ref mainHand);
                pp.ReleaseFrame();
                //Then we return
                return;
            }
            else
            {
                //Even if the game is calibrated, the player
                //can ask for re-calibration 
                calibrate(ref mainHand);
            }
        }
        else
        {
            //if one of the hands can't be retrieved
            //, the game is no more calibrated .
            calibrated = false;
        }

        pp.ReleaseFrame();
        //If the game is not calibrated at 
        //this point, we must return.
        if (!calibrated) return;

        //The next part retrieve the data that will 
        //be used to compute the rotation angles.
        float mainHandY = mainHand.positionWorld.y;
        float mainHandZ = mainHand.positionWorld.z;

        float secondaryHandY = secondaryHand.positionWorld.y;
        float secondaryHandZ = secondaryHand.positionWorld.z;

        //Then we compute and apply the rotation angles
        controlRoll(mainHandZ, secondaryHandZ);
        controlYaw(mainHandY, secondaryHandY);
        controlPitch(mainHandY, secondaryHandY);
    }
}

The checkHands(...) function checks that the hands are mapped in the right order. It is a very simple function that reverses the hand references if the mapping is not as expected.

void checkHands(ref PXCMGesture.GeoNode mainHand, ref PXCMGesture.GeoNode secondaryHand){
   if(mainHand.positionWorld.x > secondaryHand.positionWorld.x){
      PXCMGesture.GeoNode temp = mainHand;
      mainHand = secondaryHand;
      secondaryHand = temp;
   }
}

The Update() function defines the algorithm structure and includes some of the SDK calls. Let’s look in detail at the calibration function. This function verifies if the player has one thumb up on either hand. If one thumb is up, we register the hand Y value and set the calibrated value to true.

void calibrate(ref PXCMGesture.GeoNode mainHand){
   PXCMGesture.Gesture dataMain;
   PXCMGesture.Gesture dataSecondary;
   //Thumb up on the first hand ?
   if(pp.QueryGesture(
    PXCMGesture.GeoNode.Label.LABEL_BODY_HAND_PRIMARY,
    out dataMain)){
      if(dataMain.label ==
 		PXCMGesture.Gesture.Label.LABEL_POSE_THUMB_UP){
         calibrated = true;
         calibrationY = mainHand.positionWorld.y;
      }
   }else if(pp.QueryGesture(//Thumb up on the other ?
    PXCMGesture.GeoNode.Label.LABEL_BODY_HAND_PRIMARY,
    out dataSecondary)){
      if(dataSecondary.label == 
		PXCMGesture.Gesture.Label.LABEL_POSE_THUMB_UP){
         calibrated = true;
         calibrationY = mainHand.positionWorld.y;
      }
   }	
}

The last part concerns the computation of the rotation factor for each axis. The code is very close to the explanations provided in Section 4.

void controlRoll(float mainHandZ, float secondaryHandZ){
   float roll = mainHandZ - secondaryHandZ;
   //Here is the trick to smooth the moves
   roll *= Mathf.Abs (roll);
   roll *= sensibilityFactor;
   transform.RotateAroundLocal(transform.forward, roll);
}
	
void controlYaw(float mainHandY, float secondaryHandY){
   float yaw = mainHandY - secondaryHandY;
   //Here is the trick to smooth the moves
   yaw *= Mathf.Abs (yaw);
   yaw *= sensibilityFactor;
   transform.RotateAroundLocal(transform.up, yaw);
}
	
void controlPitch(float mainHandY, float secondaryHandY){
   float positionY = (mainHandY<secondaryHandY) ? 
mainHandY : secondaryHandY;
   float pitch = calibrationY - positionY;
   //Here is the trick to smooth the moves
   pitch *= Mathf.Abs (pitch);
   pitch *= sensibilityFactor;
   transform.RotateAroundLocal(transform.right, pitch);
}

In this implementation, you can see that we use the absolute value instead of the raw value. The reason is simple—we don’t want to change the sign of the angle computed.

As for,

We are doing the same thing in the code snippet:

pitch *= Mathf .Abs ( pitch ) ; // pitch = x*x
2 pitch *= sensibilityFactor ; // pitch *= 5

7 Conclusion


This paper introduced a simple method to implement smooth motion controls with the Intel Perceptual Computing SDK in a flight simulator game. It also shows the method used to find this solution.

The chosen solution has little impact on the CPU and can be adapted to other similar problems.

We’ve also seen that a simple function such as the square function does a very good job. In other situations, the Log, or square root, function can have interesting behaviors too.

An important step when looking for an algorithm is to express the problem and the solution. Once the solution is expressed, it’s easier to find a way to solve it. Here, expressing the solution was: “I am looking for a function that decreases values smaller than 0.2 and increases values greater than 0.2.”

Intel and the Intel logo are trademarks of Intel Corporation in the U.S. and/or other countries. Copyright © 2013 Intel Corporation. All rights reserved. *Other names and brands may be claimed as the property of others.

Para obtener información más completa sobre las optimizaciones del compilador, consulte nuestro Aviso de optimización.