The Intel® RealSense™ SDK has been discontinued. No ongoing support or updates will be available.
This code sample uses the Intel® RealSense™ SDK for Windows* to demonstrate some of the facial recognition capabilities of the Intel® RealSense™ user-facing camera. The SDK provides several algorithms for detecting the user’s face, facial landmark point features, head pose (roll, pitch and yaw orientation), and facial expressions.
The SDK also includes algorithms for comparing the user’s face with a set of reference images stored in a recognition database to determine the user’s identification. This feature has many potential applications in immersive gaming, security, assistive technologies, and other compelling use cases.
This code sample explores the following facial recognition actions:
- Detect the number of faces in view of the camera
- Register and unregister users
- Capture an image of the user upon registration
- Create, read, update, and delete a recognition database
The purpose of this app is to demonstrate the basics of using the face recognition capabilities of the SDK and is not intended to perform a specific task or solve a defined problem. However, by studying this example, you can decide what novel usages you can unlock by using the Intel RealSense SDK in your applications.
Figure 1. Face Recognition User Interface
The sample app displays the camera’s color stream in a WPF Image control as shown in Figure 1. When the app is first launched a database file will not be present, so the User ID indicates “Unrecognized” on the UI. The total number of faces within the field of view of the camera will be shown, but the app intentionally limits its attempts to only recognizing the first face to come into view.
Note: Recognition of multiple faces within the camera’s field of view is possible if you are developing an application that requires this feature. This is accomplished by first calling the QueryNumberOfDetectedFaces() method and then looping over calls to QueryFaceByIndex(0..n-1) to acquire and act on multiple Face instances.
Face Tracking Indicators
By default, the sample app shows a rectangular face marker that tracks the user’s face, but this can be hidden by unchecking the Show face marker checkbox. The tracking rectangle scales in size to the user’s face as he or she moves toward or away from the camera. The face marker will disappear, and the border surrounding the Image control will turn red, when the user moves out of range of the camera (Figure 2).
Figure 2. User Out of Camera View
Registering a User
Clicking the Register User button adds the user’s image to the recognition database in memory. A unique identification number is automatically assigned by the SDK, which is displayed on the screen. The ID number will also be displayed above the tracking rectangle if the face marker is activated (Figure 3).
Figure 3. User ID Registered
When an unrecognized face is registered, the app also saves a snapshot in a file named “image.jpg” that can be opened in any image viewer. This feature is included in this sample app to help with testing (Figure 4).
Figure 4. Image Capture for Testing and Experimentation
At this point a recognition database is resident in memory but not yet committed to a file on the computer’s hard drive. When the user clicks Save Database, a file named “database.bin” is created in the output folder. It is possible to save and open multiple recognition database files with any valid filename, but this sample app uses a hardcoded filename for the sake of simplicity.
Unregistering a User
A recognized user can be removed from the local database in memory by clicking the Unregister User button. This action does not commit the change to nonvolatile disk memory; to do this, the user must click the Save Database button again.
The user can also delete the database file from the computer’s hard drive by clicking the Delete Database button. This action removes the file from the hard drive but has no effect on the recognition database running in memory. When the app is restarted, the UI will indicate “Database: Deleted” and the recognition database will have to be rebuilt.
Note: The reason for these discrete steps in the sample app is to separate the various interactions into functional blocks that are easy to follow in the code. As previously mentioned, the app is not intended to perform any specific task, but instead show the basics of using the SDK for facial recognition and simple database handling.
Code Development Details
The sample app can be built using Microsoft Visual Studio* Express 2013 for Windows Desktop or the professional versions of Visual Studio 2013.
You should have some knowledge of C#, WPF, and know some of the basic operations in Visual Studio like building an executable, etc. Your system needs a front-facing 3D depth camera compatible with the Intel RealSense SDK for the example code to work correctly.
A number of private objects and member variables with global scope are declared at the beginning of the MainWindow class. These objects are instantiated and variables initialized in the MainWindow constructor.
A method named ConfigureRealSense() is called to instantiate a SenseManager object and enable the color stream, face module, 3D face tracking, and facial recognition. The code to create a recognition database is also part of this method.
A worker thread is spawned in which the acquire/release frame loop runs. As described in the Intel RealSense SDK Reference Manual, the SenseManager interface can be used in one of two ways: either by procedural calls or by event callbacks. This sample app uses procedural calls as its interfacing technique.
The Window_Closing() event handler is raised when the user closes the application by clicking the X button in the upper right-hand side of the main window. The Window_Closing() and Exit button event handlers both call a common method named ReleaseResources() that performs the necessary memory cleanup before the app closes.
User interface updates are performed in a method named UpdateUI(), which is called from within the acquire/release frame loop. UpdateUI() uses a Dispatcher.Invoke method to perform operations that will be executed on the UI thread. These operations include displaying the color stream via a WPF Image control and displaying status messages.
Saving the Database
When a user clicks the Save Database button, a Byte array is declared and dimensioned to the size of the database, which is returned by calling the QueryDatabaseSize() method. The array is then populated by passing it to the QueryDatabaseBuffer() method. Finally, the database is committed to nonvolatile disk memory by simply calling File.WriteAllBytes().
Note: The SDK documentation encourages software developers to apply industry standard encryption to protect privacy; however, file encryption techniques go beyond the scope of this introductory code sample.
Loading the Database
When the app starts, it tries to open the database file. If a file is found, its contents are read into a Byte array that gets passed to the SetDatabaseBuffer() method. The image-related content contained in the database is then used by the app for subsequent attempts at user recognition (until a new database is saved).
For this sample app, database files were tested for portability between different computers using both the integrated Intel RealSense camera and the external developer model. The results of this testing were favorable, but developers should be aware of potential differences in camera parameter settings (e.g., power, etc.) that may affect the performance of facial recognition from one camera to the next.
Check It Out
Download the app and experiment with it to learn more about how facial recognition works in the Intel RealSense SDK for Windows.
About Intel® RealSense™ Technology
To get started and learn more about the Intel RealSense SDK for Windows, go to https://software.intel.com/en-us/intel-realsense-sdk
About the Author
Bryan Brown is a software applications engineer in the Developer Relations Division at Intel.