User Segmentation

Top  Previous  Next

The User Segmentation module generates a segmented image per frame which can be used to remove or replace portions of the image behind the user's head and shoulders (background).

Input and Output (Segmented) Images

The module works on synchronized color and depth images. Multiple resolutions are supported such as [color 320x240/depth 640x480] and [color 640x360/depth 640x480]. The resolution of the output image always matches the input color image resolution. The output image is in the 32-bit BGRA (PIXEL_FORMAT_RGB32) format and contains a copy of the input color data and a synthesized alpha channel (mask). Pixels which correspond to the background will contain an alpha channel of zero, and pixels which correspond to the user will have an alpha value greater than zero.

SenseManager Programming

To activate user segmentation in the PXC[M]SenseManager multimodal pipeline, use the Enable3DSeg function. You can implement user segmentaion in your application in two ways: using SenseManager procedural functions of using SenseManager callback functions.

User Segmentation Using the SenseManager  Procedural Functions

Example 90 shows an example of a typical user segmentation application. The example does the following:

Create an instance of the PXC[M]SenseManager interface from the PXC[M]Session instance.
Activate user segmentation by using the Enable3DSeg function.
Initialize the PXC[M]SenseManager pipeline (Init) and loop through the image frames (AcquireFrame and ReleaseFrame). Whenever a segmented image is available, the Query3DSeg function returns a valid instance.

Such checking with the Query3DSeg function is optional if you call the AcquireFrame function with ifall=true, which blocks until all modality results are ready for processing.

During streaming, you may call the Pause3DSeg function to pause/resume the module operation under certain conditions.
Clean up the sample.

C++ Example 90: User Segmentation using the SenseManager Procedural Functions

// Create a SenseManager instance

PXCSenseManager *sm=PXCSenseManager.CreateInstance();

 

// Enable user segmentation

sm->Enable3DSeg();

 

// Initialize the pipeline

sm->Init();

 

// Stream data

while (sm->AcquireFrame(true)>=PXC_STATUS_NO_ERROR) {

  // Retrieve the results if ready

  PXC3DSeg *seg=sm->Query3DSeg();

  if (seg) {

      PXCImage *image=seg->AcquireSegmentedImage();

      ....

      image->Release();

  }

 

  // Resume next frame processing

  sm->ReleaseFrame();

}

 

// Clean up

sm->Release();

C# Example 90: User Segmentation using the SenseManager Procedural Functions

// Create a SenseManager instance

PXCMSenseManager sm=PXCMSenseManager.CreateInstance();

 

// Enable user segmentation

sm.Enable3DSeg();

 

// Initialize the pipeline

sm.Init();

 

// Stream data

while (sm.AcquireFrame(true)>=pxcmStatus.PXCM_STATUS_NO_ERROR) {

   // Retrieve the results

   PXCM3DSeg seg=sm.Query3DSeg();

   if (seg!=null) {

       PXCMImage image=seg.AcquireSegmentedImage();

       ...

       image.Dispose();

   }

 

   // Resume next frame processing

   sm.ReleaseFrame();

}

 

// Clean up

sm.Dispose();

Java Example 90: User Segmentation using the SenseManager Procedural Functions

// Create a SenseManager instance

PXCMSenseManager sm=PXCMSenseManager.CreateInstance();

 

// Enable user segmentation

sm.Enable3DSeg();

 

// Initialize the pipeline

sm.Init();

 

// Stream data

while (sm.AcquireFrame(true)>=pxcmStatus.PXCM_STATUS_NO_ERROR) {

   // Retrieve the results

   PXCM3DSeg seg=sm.Query3DSeg();

   if (seg!=null) {

       PXCMImage image=seg.AcquireSegmentedImage();

       ...

       image.close();

   }

 

   // Resume next frame processing

   sm.ReleaseFrame();

}

 

// Clean up

sm.close();

User Segmentation Using the SenseManager Callback Functions

You may also use the PXC[M]SenseManager callback functions. Example 91 shows the user segmentation application using callbacks. The SDK invokes the OnModuleProcessedFrame callback when there is some result for processing.

C++ Example 91: User Segmentation using the SenseManager Event Functions

class MyHandler: public PXCSenseManager::Handler {

public:

   virtual pxcStatus PXCAPI OnModuleProcessedFrame(pxcUID mid, PXCBase *module, PXCCapture::Sample *sample) {

       // check if the callback is from the user segmentation module.

       if (mid==PXC3DSeg::CUID) {

           PXC3DSeg *seg=module->QueryInstance<PXC3DSeg>();

           ...

       }

       // return NO_ERROR to continue, or any error to abort.

       return PXC_STATUS_NO_ERROR;

   }

};

 

// Create a SenseManager instance

PXCSenseManager *sm=PXCSenseManager::CreateInstance();

 

// Enable user segmentation

sm->Enable3DSeg();

 

// Initialize and stream data.

MyHandler handler;

sm->Init(&handler);

sm->StreamFrames(true);

 

// Clean up

sm->Release();

C# Example 91: User Segmentation using the SenseManager Event Functions

pxcmStatus OnModuleProcessFrame(Int32 mid, PXCMBase module, PXCMCapture.Sample sample) {

   // check if the callback is from the user segmentation module.

   if (mid==PXCM3DSeg.CUID) {

       PXCM3DSeg seg=module.QueryInstance<PXCM3DSeg>();

       ...

   }

   // return NO_ERROR to continue, or any error to abort.

   return pxcmStatus.PXCM_STATUS_NO_ERROR;

}

 

// Create the SenseManager instance

PXCMSenseManager sm=PXCMSenseManager.CreateInstance();

 

// Enable user segmentation

sm.Enable3DSeg();

 

// Initialize

PXCMSenseManager.Handler handler=new PXCMSenseManager.Handler {

   onModuleProcessedFrame=OnModuleProcessedFrame

};

sm.Init(handler);

 

// Streaming data

sm.StreamFrames(true);

 

// Clean up

sm.Dispose();

Java Example 91: User Segmentation using the SenseManager Event Functions

class MyHandler implements PXCMSenseManager.Handler {

   public pxcmStatus OnModuleProcessedFrame(int mid, PXCMBase module, PXCMCapture.Sample sample) {

       // check if the callback is from the user segmentation module.

       if (mid==PXCM3DSeg.CUID) {

           PXCM3DSeg seg=(PXCM3DSeg)module.QueryInstance(PXCM3DSeg.CUID);

           ...

       }

       // return NO_ERROR to continue, or any error to abort.

       return pxcmStatus.PXCM_STATUS_NO_ERROR;

   }

};

 

// Create the SenseManager instance

PXCMSenseManager sm=PXCMSenseManager.CreateInstance();

 

// Enable user segmentation

sm.Enable3DSeg();

 

// Initialize

MyHandler handler=new MyHandler();

sm.Init(handler);

 

// Streaming data

sm.StreamFrames(true);

 

// Clean up

sm.close();