Gesture Detector problem with custom depth image data

Gesture Detector problem with custom depth image data


I get an exception from within the core libs when using the gesture detector in combination with custom depth images.

The exception only occurs on certain combinations of confidence map values and depth values. When I set the distance far away it works (e.g. offset = 500 + some distance..) and when I exchange the confidence values it works also. Are the confidence values normalized? And what is the value for high confidence (1 or 32767)?

It seems that something is triggered within the algorithm when my hand performs a swipe action, because the exception happens reproducible more often..

I wonder also about the image options IMAGE_OPTION_NO_CONFIDENCE_MAP and IMAGE_OPTION_NO_UV_MAP. If I use the confidance map option it is still necessary to set the planes[1] pointer. So the map is used always regardless of the option parameter?

Thanks for any help on this!

This is the stack trace and code:

> CILib.dll!10064d0e()
[Frames below may be incorrect and/or missing, no symbols loaded for CILib.dll]

10064CE3 movaps xmm3,xmm0
10064CE6 mov dword ptr [esp+14h],ecx
10064CEA inc ecx
10064CEB add ebx,4
10064CEE cmp ecx,edi
10064CF0 jb 10064C70
10064CF6 mov eax,dword ptr [esp+14h]
10064CFA jmp 10064CFE
10064CFC mov eax,ecx
10064CFE mov edx,dword ptr [esp+18h]
10064D02 mov ecx,dword ptr [edx+4]
10064D05 mov eax,dword ptr [ecx+eax*4]
10064D08 mov edx,dword ptr [esi+0D0h]
10064D0E movzx edi,byte ptr [edx+eax] -> read access violation

void CMyGestureViewer::LoadImageFromDepth(PXCImage **dst_img, pxcBYTE* distData)
 IppStatus stat;
 Ipp16s max16;
 int idx;

for (int i = 0; i < m_width*m_height; i++)
 if (((unsigned short*)distData)[i] == 32767) //if it's a bad pixel
 ((unsigned short*)m_confidenceMap)[i] = 1; //set to poor? confidence
 ((unsigned short*)distData)[i] = 0; //zeroing invalid pixel for maximum calc
 ((unsigned short*)m_confidenceMap)[i] = 32767;//set to the best? value 

stat = ippsMaxIndx_16s((Ipp16s*)distData, (m_width * m_height), &max16, &idx);

for (int i = 0; i < m_width*m_height; i++)
 if (((unsigned short*)m_distData)[i] == 0)
 ((unsigned short*)m_distData)[i] = 32767; //set invalid pixels far away
 else //offset of 20cm + measured depth
 ((unsigned short*)m_distData)[i] = 100 + (max16 - ((unsigned short*)distData)[i]);

PXCImage::ImageInfo info;
 memset(&info, 0, sizeof(info));

info.height = m_height;
 info.width = m_width;
 info.format = PXCImage::COLOR_FORMAT_DEPTH;

PXCImage::ImageData data;
 memset(&data, 0, sizeof(data));
 data.format = PXCImage::COLOR_FORMAT_DEPTH;

data.planes[0] = (pxcBYTE*)m_distData;
 data.planes[1] = (pxcBYTE*)m_confidenceMap;
 data.planes[2] = 0; //no uv map
 data.pitches[0] = m_width*2; // bytes between image lines

pxcStatus sts = m_accelerator->CreateImage(&info, PXCImage::IMAGE_OPTION_NO_CONFIDENCE_MAP | PXCImage::IMAGE_OPTION_NO_UV_MAP, &data, dst_img);

BOOL CMyGestureViewer::ProcessFrame(BYTE *pData, unsigned short* distData, long lDataLen)
 PXCSmartSPArray sps(1);
 pxcStatus sts;
 PXCImage* imageDepth;

LoadImageFromDepth(&imageDepth, (pxcBYTE*)distData);

m_colorRender->RenderFrame(imageDepth, m_gestureDetector, &g_gdata, 0);

 delete imageDepth;
 return TRUE;

publicaciones de 8 / 0 nuevos
Último envío
Para obtener más información sobre las optimizaciones del compilador, consulte el aviso sobre la optimización.

is there really nobody using the confidance map or the image options up to now? Or have I missed something and my question is just too stupid?...
If there is some detailed documentation on this topic i would appriciate if anyone could give me a hint where i find it!

Hey fly,

I am having the same kind of troubles, but I think it's related to the asynchronous execution pipeline. I am not creating an PXCImage in my application, but I am attempting to develop my own pipeline classes that make more sense to me than what was included in the SDK.

The order in which you call ProcessImageAsync, and when/how you synchronize with the SyncPoint it returns is the part I don't fully understand yet.

From my tests, the confidence map is required by the gesture detection algorithm. I created a PXCCapture::VideoStream with the imageOptions set to exclude the confidence map, and it failed immediately. With the confidence map enabled however, it continues until I capture imagery with anything close to the camera (1-2 feet). That's when the gesture detection fails for me. So that sounds like the same issue with the confidence map.

I think it has to do with the execution pipeline because my code is creating it's own pipeline, and your code is altering the pipeline (intentionally or otherwise) as well.

Hope that helps,


Hi David,

thank you, it helps a bit, at least I know now that I'm not alone facing this problem :-)

The only difference I see is that you say the detection fails at any close object - for me it fails only when I do movements in a close range.

Hope I find some time soon to figure out what's going on here...

Now I had time to verify the PXCSDK again. Trying a different attempt, using the gesture_viewer sample and the libpcxutils source, same results.

The renderer works fine, displays the depth map and the confidance map, but as soon I activate the gesture module the same crash happens as with my custom pipeline. So both, my custom and the pipeline from the utils lib are failing and obviously it is not possible with this SDK to make any change in the signal chain before the output of the gesture module. A big undocumented blackbox that throws exceptions when you touch it too much...

Sorry about the undocumented features. There are three things here:

(1) The finger tracking module requires depth data and confidence data. So you must at least provide some fake confidence data.

(2) The ProcessImageAsync function takes a image array PXCImage[4]. This is corresponding to what you have got back from QueryProfile. The ProfileInfo.inputs fields tell you exactly what the module needs, its color format.

(3) The module also needs certain device properties to function. They are part of the ProfileInfo.inputs description. If you don't have a camera, then you have to provide something equivelant.

In summary, the input need of a module is explicitly described when you call the QueryProfile function. This includes the device properties and the stream descriptions.


Thanks for the help! Finally I got the whole thing running, using custom confidance data and so. It helped a lot to investigate the data stream of the camera.

The root cause of my problems was that I called the gesture detector assignments before I called the LocateStreams method. It seems that somehow this causes the module to crash whenever a gesture would have been detected.

By now I am running a custom depth sensing device without much preprocessing of data (only smoothing) and get already very good results regarding detection accuracy although my custom light source is worse than the original one. So I think the performance of the SDK is also limited by the camera performance and I'll give it a try to improve it, because the SDK it doing a great job, but to satisfy the standard user it might be necassary that it perform a little bit better.

Yes, Location Stream does two things:

(1) Initialize the input streams.

(2) Set the device properties (and stream parameters) required by the gesture module.

That's why running the detector before LocateStream will likely crash the system.

Glad that you find the right workaround.

Deje un comentario

Por favor inicie sesión para agregar un comentario. ¿No es socio? Únase ya