Archived - Background Replacement Code Sample Using Intel® RealSense™ Technology

The Intel® RealSense™ SDK has been discontinued. No ongoing support or updates will be available.

Download Code Sample

Abstract

This code sample demonstrates how to create a real-time “green screen” effect for capturing augmented selfies using the Intel® RealSense™ SDK for Windows*. The SDK includes a User Segmentation module that generates a segmented image on each frame that can be used to remove or change the background behind the user’s body. This sample app demonstrates a method for replacing the background with a static image selected by the user.

Background Replacement Sample

Figure 1. User Segmentation, Before and After

Introduction

On startup the app replaces the live background behind the user with a solid green image. The selection of green for the initial bitmap background color is completely arbitrary. It is simply a nod to the traditional chroma key compositing (or “green screen”) technique for removing the background from the subject of a photograph or video.

In contrast to chroma keying and 2D camera background subtraction techniques, the User Segmentation module works on synchronized color and depth images streaming from the 3D camera. Unlike 2D methods, there is no possibility of colors on the user’s clothing being interpreted as part of the background, or of a moving background being interpreted as part of the user.

The module outputs a 32-bit BGRA formatted image consisting of the input color data and a synthesized alpha channel (referred to as a mask). Pixels corresponding to the background have an alpha channel of zero, and pixels corresponding to the user will have an alpha value greater than zero. (Refer to the installed SDK manual located in RSSDK_DIR\doc\PDF\sdkmanuals.pdf for more details.)

The app includes a Checkbox control that allows the user to remove the background image if desired (see the “before” image in Figure 1). Background images are selected by clicking the “Open File” button, which opens a dialog box allowing the user to navigate to image files (*.bmp, *.jpg, *.png, *.gif) on the hard drive (Figure 2).

Open File Dialog Box

Figure 2. Open File Dialog Box

The selected background image is automatically scaled by the app to the pixel dimensions of the color and depth streams (640x480). On each frame the background and segmented images are stitched together and rendered in a WPF Image control.

When the “Capture” button is clicked, a snapshot is taken and saved to the output folder as “MyPic.jpg” (Figure 3).

Captured Image

Figure 3. Captured Image

The Depth Advantage

The Intel® RealSense™ 3D Camera (F200) houses both a 640x480 resolution IR camera and a 1080p RGB color camera. Combining data from both the IR depth and color cameras has the advantage of overcoming many of the issues associated with 2D camera background subtraction, such as a dynamic backgrounds (e.g., people or objects moving in the background), shadows, and varying lighting conditions.

Development Environment

The sample app can be built using Microsoft Visual Studio* Express 2013 for Windows Desktop or the professional versions of Visual Studio 2013.

About the Code

The code sample is a Windows Desktop app coded in C# using Extensible Application Markup Language (XAML) to define the user interface. It uses the unsafe keyword to define a section of code for working with pointers to facilitate faster image processing (refer to the C# Examples website for other fast image processing techniques).

As stated in the MSDN* Unsafe Code and Pointers C# Programming Guide, “unsafe code in C# is not necessarily dangerous; it is just code whose safety cannot be verified by the CLR. The CLR will therefore only execute unsafe code if it is in a fully trusted assembly.” To use the unsafe keyword in your code you must check the “Allow unsafe code” setting under ProjectPropertiesBuild.

This project uses an explicit path to libpxcclr.cs.dll (the managed DLL): C:\Program Files (x86)\Intel\RSSDK\bin\x64. This reference will need to be changed if your SDK installation path is different. Since we are referencing the 64-bit version of the DLL, you must also ensure that “x64” is specified under the ProjectPropertiesPlatform target setting.

For more information on using the User Segmentation module please refer to the SDK’s installed documentation (RSSDK_DIR\doc\PDF\sdkmanuals.pdf) and the online Background Segmentation Tutorial.

About Intel® RealSense™ Technology

To get started and learn more about the Intel RealSense SDK for Windows, go to https://software.intel.com/en-us/intel-realsense-sdk

About the Author

Bryan Brown is a software applications engineer in the Developer Relations Division at Intel. 

For more complete information about compiler optimizations, see our Optimization Notice.

6 comments

Top

Questions regarding Intel® RealSense™ Technology and Intel® RealSense™ Developer Kits should be posted here:

https://communities.intel.com/community/tech/realsense

Questions posted to Articles may or may not be responded to.

Regards, Hal

Intel(R) Developer Zone Support

https://software.intel.com
*Other names and brands may be claimed as the property of others.

 

Hi Bryan,

I have recently involved for a project but I have faced a problem with SR300 camera and SDK R2- 2016 in the below code:

   while (senseManager.AcquireFrame(true) >= pxcmStatus.PXCM_STATUS_NO_ERROR)

Every time it equal false and the program does not work.

could you kindly help me?

many thanks

 

Hi Bryan,

We need to remove the background using depth information. We have a couple R200 cameras that we actually bought for doing this.

In the SDK samples we see that FF_3DSeg and FF_3DSeg.cs are actually available only for the other cameras.. F200 and SR300...

I don't understand why we can't use that functionalities in the R200 if we have depth streams as well as color streams on this device...

What would you recommend to perform background removal with R200 cameras?

Thank you very much!

 

rishabh_banga@yahoo.in's picture

Hi Bryan, 

Thank you for your response. I was actually able to fit 3 people, but again with their entire hairline becoming invisible. I had another query though. How do I achieve a resolution of 1080 pixels in the pictures captured? 

Hi Dave and Rishabh,

Thanks for the positive feedback on this code sample, I'm glad to hear people are finding it useful!

Regarding mapping two people at the same time, the RSSDK segmentation module is intended for a single user and no objects (although you may occasionally experience some "breakthrough" when another person enters the scene on the same depth plane as the main user).

-Bryan

Hi Bryan.  Thanks so much for this sample!  I Love It.  So straight forward.  So useful.  Really a great job.

Dave Carlson

rishabh_banga@yahoo.in's picture

Hi Bryan, thanks for this snippet of code, really helped me out with a project. Although there is one bug that I came across, and that was that the camera is not able to completely map 2 people at the same time. It completely fixates on one, which leads to some part of the other person's face to get completely ignored. Let me know if you encounter the same issue.

Add a Comment

Have a technical question? Visit our forums. Have site or software product issues? Contact support.