The Intel® RealSense™ SDK has been discontinued. No ongoing support or updates will be available.
This code sample uses the Intel® RealSense™ SDK for Windows* to create a simple virtual drawing app named Sketch. This Windows desktop app, developed in C#/WPF, demonstrates a number of hand tracking and gesture recognition capabilities available in the Intel RealSense SDK:
- Retrieval of hand joint x-y-z (world) coordinates
- Selectively capturing gestures data
- Getting alert status information (i.e., hand detection, calibration, and boundary status)
(Note: The full functionality of this sample app requires a front-facing 3D camera.)
Introduction to Sketch
Sketch is a simple drawing app that allows the user to simulate drawing on a canvas by using gestures and hand motions. Figure 1 shows the Sketch user interface (developed in WPF/XAML).
Figure 1. Sketch User Interface
Three gestures (depicted on the screen with their accompanying action) are enabled for interacting with the virtual canvas:
- Finger Pinch (“Draw”) – causes the cursor to become solid and draw a line on the canvas. The position of the cursor on the canvas is controlled by the x and y coordinates of the user’s middle fingertip. The line thickness is controlled by the z-axis of the user’s middle fingertip (i.e., pulling away from the camera causes the line to become thinner, as if decreasing pressure on the pen or brush).
- Spread Fingers (“Hover”) – deactivates the pen, making the cursor become an empty circle. Hovering allows the pen to be moved to other sections of the canvas without drawing a line. It also allows the user to select colors from the right-hand palette by simply hovering over them.
- Wave (“Erase”) – Wipes the drawing canvas clean, making it ready for a new drawing.
The Sketch app simulates drawing on a canvas when the user makes the "
two_fingers_pinch_open" gesture. This gesture was selected because it approximates the hand pose a user would make when holding a pen or brush. Figure 2 demonstrates this gesture.
Figure 2.Drawing Gesture
Pen placement and stroke thickness are determined by tracking the tip of the user’s middle finger, which may seem unintuitive considering the finger-pinch is the enabling gesture for drawing on the canvas. The reason for tracking the middle finger is to avoid occlusions that can occur when the thumb and index finger are pressed together. Tracking the middle finger instead of the index finger or thumb results in better performance.
The Sketch app also demonstrates how to acquire and display hand alert status information (in this case hand detection, calibration, and border exceptions). Providing this type of feedback, in one form or another, assists users in positioning their hands correctly in front of the camera. Although the presentation of this information is simplistic in this sample app, developers are encouraged to provide similar cues to their users to improve the overall user experience.
Check It Out
To experiment with this app and learn more about how it works, please download it here.
About Intel® RealSense™ Technology
To get started and learn more about the Intel RealSense SDK for Windows, go to https://software.intel.com/en-us/realsense/intel-realsense-sdk-for-windows.
About the Author
Bryan Brown is a software applications engineer in the Developer Relations Division at Intel. His professional experience includes a mix of software, electronic, and systems design engineering. His technical interests focus on applications of natural interaction and brain-computer interface technologies, with active participation in several alpha developer programs for various emerging technologies in these areas.