The Intel® RealSense™ SDK has been discontinued. No ongoing support or updates will be available.
This blog is meant to provide information, suggestions, and links for developers using Unity with the Intel RealSense SDK 2014. Please feel free to add info or corrections in the comments and I will add those into this article in future versions. Check back often for additions.
Important Note on Unity Versions:
- Unity 4.x is 32-bit based. You can use Unity Pro or the free version of Unity as long as you use the signed libraries in the Intel RealSense SDK R2 release (v4.)
- Unity 5 is 64-bit based. You must copy over the 64 bit libraries ( libpxccpp2c.dll and libpxccpp2c.dll.signtaure) to your project.
Documentation that may be helpful.
- Intel provided: The RSSDK Unity Toolkit guide is installed at C:\Program Files (x86)\Intel\RSSDK\doc\PDF\sdktoolkit.pdf
- Face Tracking tutorial includes face location, pose, landmark, and expression detection, face recognition, alerts and code samples
- Hand/Joint/Blob Tracking tutorial covers gesture, hand and joint tracking, blob and contour extraction, data smoothing & code samples
- Raw Stream Capture tutorial :Catching and rendering color and depth streams with code samples
- Unofficial - Intel developer Amit M has Unity Toolkit Demo and RS/Unity RFBDepth Alignment videos
- Marty G of Sambiglyon has shared these guides:
- Setting up the SDK files in Unity and programming objects in the 'Inspector' panel to respond to camera inputs.
- Converting PXC scripts for use in Unity script files
- Prototyping RealSense project controls with physical controllers using Unity's built-in "Input Manager".
- Simple Hand Tracking /Object moving guide.
It should be noted that one can use the Unity DLL without using the Unity Toolkit. If you post questions or issues, please distinguish between the two.
Some Minimum Requirements:
- With Gold R1 ONLY Windows 8.1 is supported. The only camera supported is Creative RealSense F200 USER facing camera, which must be in a true (blue) USB 3.0 direct port (not hubbed).
- NOTE: Gold R1 was compiled against Unity Pro 4.1 and was also tested on v4.5 It was not validated against 4.6 Beta.
Now some tips/tricks:
- Full Screen: Make the App go Full Screen in windowed/unwindowed modes, whatever the display resolution.
- Note Intel is suggesting a minimum resolution of 1368x768
- Before compiling, in Player’s Settings, make sure the "Use Direct3D 11" box is checked.
- If only D3D9 is available in Player Settings, Uncheck "Default is Native Resolution" or "Default is Full Screen" and set default screen width and height to 1368x768
- Set it to use ONLY specific aspect ratios: 16:9 and 16:10
- Result: Based on the "Windowed" Checkbox setting on the "Resolution Selection Screen"
- If checked, App will start windowed but can be resized/maximized
- If unchecked, App will always start at full screen.
- Note you can check resolution // Cap screen resolution, an example
if ( Screen.currentResolution.width > 1366 || Screen.currentResolution.height > 768 ) Screen.SetResolution(1366, 768, true);
The AcquireFrame desired setting in the Unity Update() function should be (false,0) to avoid impacting performance. (In Unity, it is a blocking call which locks the frame rate to RS and is used to wait until any frame data is available)
if (sm.AcquireFrame (false) != pxceStatus.PXCM_STATUS_NO_ERROR)
While 2 hands tracking may be in the same thread, it is suggested you create separate thread for face and voice or any multi-modal use.
- Smoothing in Unity: as in the SDK, 4 types of smoothing are available. Here's SUGGESTIONS for their use in Unity. Top 2 most preferred.
- Spring: Time based but has a linear effect on smoothed data (Marty suggests smoothing factor=1 with TrackingAction script.
- Quadratic: Time based smoothing which acts as stabilizer and smoother.
- Weighted: Allows averaging the data according to any set of weights (Gaussian, Uniform, etc.) . Good for hands
- Stabilize smoothing: good for UX as it keep existing location unless a large movement is detected
- Smoothing out jerky points in a camera-driven Unity sphere object (that is used as a rotational joint). If you find that the rotation is smooth in some parts of the object's rotation cycle but then suddenly jumps at a certain point, check the 'Scale' values in your sphere's settings in the Inspector panel of Unity. (Thanks Marty)
- Voice/Speech in Unity. Unity can do Commands, and Speech to text (can use the C# wrapper sample from framework folder) Speech synthesis may be doable the same way (use sample from C#) .
- 3D Segmentation - Beards. Currently long beards may cause very slow detection as the beard absorbs a lot of IR and may separate the head from body, confusing detection. The issue is being worked.
- Face Recognition Registration.
- In Gold R1, 2D face recognition is only using ID#100.
- Gold changed from Beta and now waits 10 consecutive frames before confirming recognition (not settable since used to lessen false positives.
Hand Tracking if out of FOV: Use the “HandLost” rule in the Unity Toolkit. (RSUnityToolkit/Internals/Rules). Checks for number of hands detected (SenseToolkitManager.Instance.HandDataOutput.QueryNumberOfHands())
If you have issues with UnityEngine.Texture2D PXCMImage.ImageData.ToTexture2D(Int32 plane, Int32 width, Int32 height)),
use ToTexture2D that specifies a Texture2D as a parameter.
From Marty G: If you find that your objects are moving jerkily and unpredictably or tearing themselves away from objects that they are attached to as child objects, go into the Inspector panel settings for a particular TrackingAction script and set all 'Virtual World Box' and 'Real World Box' values to zero.
From Marty G: If you want Unity to run a script inside an object other than MovingObject, edit the script inside TriggerObject to give the name of that object, the name of the script and the function type at the start of that script (Start(), Update(), etc).. For more info see this forum post.
Looking forward to hearing other tips and tricks! Will hopefully update this weekly!