Archived - Background Replacement Code Sample Using Intel® RealSense™ Technology

By Bryan K. Brown,

Published:05/04/2015   Last Updated:05/04/2015

The Intel® RealSense™ SDK has been discontinued. No ongoing support or updates will be available.

Download Code Sample

Abstract

This code sample demonstrates how to create a real-time “green screen” effect for capturing augmented selfies using the Intel® RealSense™ SDK for Windows*. The SDK includes a User Segmentation module that generates a segmented image on each frame that can be used to remove or change the background behind the user’s body. This sample app demonstrates a method for replacing the background with a static image selected by the user.

Background Replacement Sample

Figure 1. User Segmentation, Before and After

Introduction

On startup the app replaces the live background behind the user with a solid green image. The selection of green for the initial bitmap background color is completely arbitrary. It is simply a nod to the traditional chroma key compositing (or “green screen”) technique for removing the background from the subject of a photograph or video.

In contrast to chroma keying and 2D camera background subtraction techniques, the User Segmentation module works on synchronized color and depth images streaming from the 3D camera. Unlike 2D methods, there is no possibility of colors on the user’s clothing being interpreted as part of the background, or of a moving background being interpreted as part of the user.

The module outputs a 32-bit BGRA formatted image consisting of the input color data and a synthesized alpha channel (referred to as a mask). Pixels corresponding to the background have an alpha channel of zero, and pixels corresponding to the user will have an alpha value greater than zero. (Refer to the installed SDK manual located in RSSDK_DIR\doc\PDF\sdkmanuals.pdf for more details.)

The app includes a Checkbox control that allows the user to remove the background image if desired (see the “before” image in Figure 1). Background images are selected by clicking the “Open File” button, which opens a dialog box allowing the user to navigate to image files (*.bmp, *.jpg, *.png, *.gif) on the hard drive (Figure 2).

Open File Dialog Box

Figure 2. Open File Dialog Box

The selected background image is automatically scaled by the app to the pixel dimensions of the color and depth streams (640x480). On each frame the background and segmented images are stitched together and rendered in a WPF Image control.

When the “Capture” button is clicked, a snapshot is taken and saved to the output folder as “MyPic.jpg” (Figure 3).

Captured Image

Figure 3. Captured Image

The Depth Advantage

The Intel® RealSense™ 3D Camera (F200) houses both a 640x480 resolution IR camera and a 1080p RGB color camera. Combining data from both the IR depth and color cameras has the advantage of overcoming many of the issues associated with 2D camera background subtraction, such as a dynamic backgrounds (e.g., people or objects moving in the background), shadows, and varying lighting conditions.

Development Environment

The sample app can be built using Microsoft Visual Studio* Express 2013 for Windows Desktop or the professional versions of Visual Studio 2013.

About the Code

The code sample is a Windows Desktop app coded in C# using Extensible Application Markup Language (XAML) to define the user interface. It uses the unsafe keyword to define a section of code for working with pointers to facilitate faster image processing (refer to the C# Examples website for other fast image processing techniques).

As stated in the MSDN* Unsafe Code and Pointers C# Programming Guide, “unsafe code in C# is not necessarily dangerous; it is just code whose safety cannot be verified by the CLR. The CLR will therefore only execute unsafe code if it is in a fully trusted assembly.” To use the unsafe keyword in your code you must check the “Allow unsafe code” setting under ProjectPropertiesBuild.

This project uses an explicit path to libpxcclr.cs.dll (the managed DLL): C:\Program Files (x86)\Intel\RSSDK\bin\x64. This reference will need to be changed if your SDK installation path is different. Since we are referencing the 64-bit version of the DLL, you must also ensure that “x64” is specified under the ProjectPropertiesPlatform target setting.

For more information on using the User Segmentation module please refer to the SDK’s installed documentation (RSSDK_DIR\doc\PDF\sdkmanuals.pdf) and the online Background Segmentation Tutorial.

About Intel® RealSense™ Technology

To get started and learn more about the Intel RealSense SDK for Windows, go to https://software.intel.com/en-us/intel-realsense-sdk

About the Author

Bryan Brown is a software applications engineer in the Developer Relations Division at Intel. 

Product and Performance Information

1

Intel's compilers may or may not optimize to the same degree for non-Intel microprocessors for optimizations that are not unique to Intel microprocessors. These optimizations include SSE2, SSE3, and SSSE3 instruction sets and other optimizations. Intel does not guarantee the availability, functionality, or effectiveness of any optimization on microprocessors not manufactured by Intel. Microprocessor-dependent optimizations in this product are intended for use with Intel microprocessors. Certain optimizations not specific to Intel microarchitecture are reserved for Intel microprocessors. Please refer to the applicable product User and Reference Guides for more information regarding the specific instruction sets covered by this notice.

Notice revision #20110804