Archived - Tips and Tricks to use Unity* with the Intel® RealSense™ SDK 2014

The Intel® RealSense™ SDK has been discontinued. No ongoing support or updates will be available.

This blog is meant to provide information, suggestions, and links for developers using Unity with the Intel RealSense SDK 2014. Please feel free to add info or corrections in the comments and I will add those into this article in future versions. Check back often for additions.

Important Note on Unity Versions:

  • Unity 4.x is 32-bit based. You can use Unity Pro or the free version of Unity as long as you use the signed libraries in the Intel RealSense SDK R2 release (v4.)
  • Unity 5 is 64-bit based. You must copy over the 64 bit libraries ( libpxccpp2c.dll and libpxccpp2c.dll.signtaure) to your project. 

Documentation that may be helpful.

It should be noted that one can use the Unity DLL without using the Unity Toolkit. If you post questions or issues, please distinguish between the two. 

Some Minimum Requirements:

  • With Gold R1 ONLY Windows 8.1 is supported. The only camera supported is Creative RealSense F200 USER facing camera, which must be in a true (blue) USB 3.0 direct port (not hubbed). 
  • NOTE: Gold R1 was compiled against Unity Pro 4.1 and was also tested on v4.5 It was not validated against 4.6 Beta.

Now some tips/tricks:

  • Full Screen: Make the App go Full Screen in windowed/unwindowed modes, whatever the display resolution. 
    • ​Note Intel is suggesting a minimum resolution of 1368x768
    • Before compiling, in Player’s Settings, make sure the "Use Direct3D 11" box is checked. 
      • If only D3D9 is available in Player Settings, Uncheck "Default is Native Resolution" or "Default is Full Screen" and set default screen width and height to 1368x768
    • Set it to use ONLY specific aspect ratios: 16:9 and 16:10
    • Result: Based on the "Windowed" Checkbox setting on the "Resolution Selection Screen"
      • If checked, App will start windowed but can be resized/maximized
      • If unchecked, App will always start at full screen.
    • Note you can check resolution    // Cap screen resolution, an example
      if ( Screen.currentResolution.width > 1366 || Screen.currentResolution.height > 768 )             Screen.SetResolution(1366, 768, true);

  • Performance:

    • The AcquireFrame desired setting in the Unity Update() function should be (false,0) to avoid impacting performance. (In Unity, it is a blocking call which locks the frame rate to RS and is used to wait until any frame data is available)
      if (sm.AcquireFrame (false) != pxceStatus.PXCM_STATUS_NO_ERROR)

    • While 2 hands tracking may be in the same thread, it is suggested you create separate thread for face and voice or any multi-modal use.

  • Smoothing in Unity: as in the SDK, 4 types of smoothing are available. Here's SUGGESTIONS for their use in Unity. Top 2 most preferred.
    • Spring: Time based  but has a linear effect on smoothed data (Marty suggests smoothing factor=1 with TrackingAction script.
    • Quadratic: Time based smoothing which acts as stabilizer and smoother.
    • Weighted: Allows averaging the data according to any set of weights (Gaussian, Uniform, etc.) . Good for hands
    • Stabilize smoothing: good for UX as it keep existing location unless a large movement is detected
  • Smoothing out jerky points in a camera-driven Unity sphere object (that is used as a rotational joint).  If you find that the rotation is smooth in some parts of the object's rotation cycle but then suddenly jumps at a certain point, check the 'Scale' values in your sphere's settings in the Inspector panel of Unity. (Thanks Marty)
  •  
  • Voice/Speech in Unity. Unity can do Commands, and Speech to text (can use the C# wrapper sample from framework folder)  Speech synthesis may be doable the same way (use sample from C#) . 
     
  • 3D Segmentation - Beards. Currently long beards may cause very slow detection as the beard absorbs a lot of IR and may separate the head from body, confusing detection. The issue is being worked.
  •  
  • Face Recognition Registration.
    • In Gold R1, 2D face recognition is only using ID#100. 
    • Gold changed from Beta and now waits 10 consecutive frames before confirming recognition (not settable since used to lessen false positives.
       
  • Hand Tracking if out of FOV: Use the “HandLost” rule in the Unity Toolkit. (RSUnityToolkit/Internals/Rules). Checks for number of hands detected (SenseToolkitManager.Instance.HandDataOutput.QueryNumberOfHands()) 

  • If you have issues with UnityEngine.Texture2D PXCMImage.ImageData.ToTexture2D(Int32 plane, Int32 width, Int32 height)),
    use
    ToTexture2D that specifies a Texture2D as a parameter.

  • From Marty G: If you find that your objects are moving jerkily and unpredictably or tearing themselves away from objects that they are attached to as child objects, go into the Inspector panel settings for a particular TrackingAction script and set all 'Virtual World Box' and 'Real World Box' values to zero.

  • From Marty G: If you want Unity to run a script inside an object other than MovingObject, edit the script inside TriggerObject to give the name of that object, the name of the script and the function type at the start of that script (Start(), Update(), etc).. For more info see this forum post.

  •  

Looking forward to hearing other tips and tricks!  Will hopefully update this weekly!

For more complete information about compiler optimizations, see our Optimization Notice.

3 comments

Top
Kiriakos T.'s picture

Using the RealSemse SDK 2015 R5 for Intel(R) RealSense (TM) 3D Camera (Front F200) in Unity (version 5.3.1f1) I have encountered serious problems as far as the precision of the hand tracking is concerned. In every little move of the hand, the join between a certain part of the hand and an object that is adjusted on it is lost. It works fine when the hand is standstill but not when it moves. I tried to do some correction in unity 's inspector options but I didn't make sufficient improvement. Is there anything I can try to have better results?

Anthony M.'s picture

I really like those tricks. My usb port is not USB 3.0 direct port (not hubbed) so i have to check on that.

Marty G.'s picture

Thanks so much for the excellent advice, Colleen!  My company is continuing to develop its contest entry for the next few months beyond the contest end to make it commercially releasable and improving our Unity RealSense tech all the time as we do so (we will keep sharing these new techniques with everyone.)  .  

Once you invest time in learning the alchemy between RealSense and Unity and finding new and totally unexpected ways to make them work together better, they really do become a great combination that can produce motion capture results that used to cost companies many thousands with custom-built hardware. It still surprises me every day how much game development power can be harnessed from the camera when you keep pushing it further.

Add a Comment

Have a technical question? Visit our forums. Have site or software product issues? Contact support.