Conversion from Depth to World Coordinates

Conversion from Depth to World Coordinates

I would like to generate point clouds from the sensor depth data, so I would like to know if there is a way of obtaining the world coordinates from the depth pixels. I've searched the APIs but I haven't found an obvious way to do this. Has anyone found a solution to this? 

17 posts / 0 new
Last post
For more complete information about compiler optimizations, see our Optimization Notice.

The camera can output depth data in different formats. In PXCImage::COLOR_FORMAT_VERTICES format, the first plane will contain three-channel (X,Y,Z) image with world coordinates.
You can easily request this format using UtilCapture class:


    PXCCapture::VideoStream::DataDesc request;

    memset(&request, 0, sizeof(request));

    request.streams[0].format=PXCImage::COLOR_FORMAT_VERTICES;

    capture.LocateStreams (&request);


More details from sdk-manual-core:
PXCImage::COLOR_FORMAT_DEPTH: The depth map, which contains three planes, unless otherwise specified in ImageOption. The first plane contains a depth map; the second plane contains the corresponding confidence map; and the third plane contains the corresponding UV map.
PXCImage::COLOR_FORMAT_VERTICES: The fixed-point vertices map, which contains three planes, unless otherwise specified in ImageOption. The first plane contains the vertices in fixed-point integers; the second plane contains the corresponding confidence map; and the third plane contains the corresponding UV map.

Thanks that is very helpful! Now the problem is how to extract the vertex image correctly. I am using UtilCapture and initialize the streams like so


       PXCCapture::VideoStream::DataDesc request;

	memset(&request, 0, sizeof(request));

	request.streams[0].format=PXCImage::COLOR_FORMAT_RGB32;			// color image

	request.streams[1].format=PXCImage::COLOR_FORMAT_DEPTH;			// depth map

	request.streams[2].format=PXCImage::COLOR_FORMAT_VERTICES;		// depth converted to world coordinates

	capture.LocateStreams (&request);

However when retrieving the images, I do not get the correct image type for the world coordinate stream:


   rgbImage = capture.QueryImage(images,PXCImage::IMAGE_TYPE_COLOR);

    depthImage = capture.QueryImage(images,PXCImage::IMAGE_TYPE_DEPTH);

    vertImage = capture.QueryImage(images, PXCImage::IMAGE_TYPE_DEPTH & PXCImage::COLOR_FORMAT_VERTICES);

When querying it, vertImage retains the same image type as depthImage, i.e., COLOR_FORMAT_DEPTH.

Am I passing the wrong arguments to capture.QueryImage() for vertimage. The documentation does recommed applying the PXCImage::IMAGE_TYPE_DEPTH & PXCImage::COLOR_FORMAT_VERTICES mask.

I just noticed when I do not request both the COLOR_FORMAT_DEPTH and COLOR_FORMAT_VERTICES but only the vertices stream, I obtain the vertex type image. Is it impossible to retrieve both at the same time?

Hi Sven,

Have got the the point cloud successfully? I'm a rookie in both programming and this sdk, I'll be appreciated if you can answer a simple question.

How did you save the vertImage? In my program I put the line "vertImage = capture.QueryImage(images, PXCImage::IMAGE_TYPE_DEPTH & PXCImage::COLOR_FORMAT_VERTICES);" but it doesn't work. In your previous post you only showed part of the codes, do you mind also posting the missing part?

Thanks in advance.

Johnny

Hi Johnny X,

here is the main loop that extracts RGB and vertimage. Unfortunately, it seems impossible to extract the depth image as well as the vertex image at the same time. Intel has not yet answered me on this issue.


  // Create session

    PXCSmartPtr session;

    pxcStatus sts=PXCSession_Create(&session);

    if (sts images(2);

                PXCSmartSPArray sps(1);

                pxcStatus sts=capture.ReadStreamAsync(images,&sps[0]);
		if (IsCameraDisconnected(sts)) continue; // device disconnected

		if (stsSynchronize(0);
		if (IsCameraDisconnected(sts)) continue; // device disconnected

		if (sts::Ptr PCLCloudPtr ;
	PXCImage * vImage = vertImage;
        // acquiure access to depth data

	PXCImage::ImageData ddata;

	vImage->AcquireAccess(PXCImage::ACCESS_READ, &ddata); 
        short * vertices = (short*) ddata.planes[0];
        // get image information

	pxcStatus stat = vImage->QueryInfo(&vertImageInfo);
	const int w = vertImageInfo.width;

	const int h = vertImageInfo.height;
         // see typedefs

	cloud = CloudPtr (new pcl::PointCloud(w,h));
	for (int i = 0; i < w; i++)

	{

		for (int j = 0; j < h; j++)

		{

			unsigned int idx = 3*(j*w + i);  // three coordinates per point
			short x = vertices[idx];

			short y = vertices[idx+1];

			short z = vertices[idx+2];
			if (z < 2000)

			{

                                // normalize vertices to meters

				cloud->points[idx/3].x = ((float) x) / 1000.0;

				cloud->points[idx/3].y = ((float) y) / 1000.0;

				cloud->points[idx/3].z = ((float) z) / 1000.0;

			}
		}

	}

Hi Sven,
Thanks for the prompt reply! Now I can make the 1st part working properly.

For the 2nd part using PCL, what is the definition of "&vertImageInfo" in line16 and "cloud" in line22 ?
It would be helpful if you could also show the header files that are included.

Thanks,
Johnny

Johhny X., vertImagInfo is a PXCImage::ImageInfo object, which is documented in the framework.

Hi,
Am working on gesture Recognition, trying to bypass the camera and identify the gesture from a single image.
Am giving input ,a depth image(Format - bitmap) and converted that into PXC Image.

1. Will the depth details are available in the converted PXC Image.
2. I want to Process this single image and get the blob image and data out of it using Processimageasync. whether it is possible?

now iam facing one problem in processimageasync,it is throwing exception like unhandled exception in gesture_new.exe:0x00000005:Access vioalation reading location 0x00000000
iam getting pitches and planes values as 0..can u plz suggest what i need to do

Thanks in Advance

Hi Sven,

I've been having a play as I want to do something very probably very similar to you and hit the issues you have described. I believe you are able to get depth and vertices at the same time but I think there is a limit in the SDK on the number of streams. This means you can have a maximum of two simultaneous video streams so you can have depth and vertices or RGB and depth or YUY and vertices and so on.

Can anyone from Intel confirm?

David

Ok, I've had a bit more of a play and I can get RGB24, Depth, Vertices and PCM data being captured and rendered via Capture_Viewer app so the SDK can support it. However, all streams must be on the same logical device i.e. DepthSense Device 325V2 so you cannot get depth from this and color stream data from Creative GestureCam.

Also depth data streams at fastest rate selected so if Depth 320x240 30fps and Vertices 320x240 60fps both stream at 60fps.

David

@daflippers could you post some code to show how you did it? I don't fully understand what you mean about logical devices. I'm only using a single camera with a single application instance accessing it, so in theory only one logical device (?). Does UtilCapture open more than one logical device?

Hi Sven,

Sorry, I am playing catch up as the email subscription has been turned off due to Spammers so I understand.

I used the capture_viewer sample to select the various streams available from the 325V2 camera and you can simultaneously display the streams as I posted above so it's worth checking out the sample.

 

David

I've been trying to get vertices and their color together. Does anyone have any code or a link to an example for this? I've been struggling to get it working for a couple days. I want to visualize it using PCL. I've been able to get the vertices. But I can't seem to get the color data properly. I've been trying to modify the sample code from the UV Map sample that comes with the SDK.

To get color and vertics together, you can refer to the camera_uvmap sample.

Change the line:

    request.streams[1].format=PXCImage::COLOR_FORMAT_DEPTH;

to be:

    request.streams[1].format=PXCImage::COLOR_FORMAT_VERTICES;

You can then get the streams. Get rid of the rest code that does projection (which relies on the depth data.)

 

Following up last year's post...I'm just getting back into developing a further application using the Intel Perceptual SDK. I now need access to the camera's depth stream, and want to convert the depth images to an OpenCV Mat. 

The documentation is IMHO unclear about the exact format of the depth stream:

The depth map, which contains three planes, unless 
otherwise specified in ImageOption. The first plane 
contains a depth map; the second plane contains the 
corresponding confidence map; and the third plane 
contains the corresponding UV map. 

What format is the depth map, and confident map. Is it 16 bit unsigned? I've seen examples where floats are extracted from the depth stream, so I'm a bit confused. 

 

 

Following up last year's post...I'm just getting back into developing a further application using the Intel Perceptual SDK. I now need access to the camera's depth stream, and want to convert the depth images to an OpenCV Mat. 

The documentation is IMHO unclear about the exact format of the depth stream:

The depth map, which contains three planes, unless 
otherwise specified in ImageOption. The first plane 
contains a depth map; the second plane contains the 
corresponding confidence map; and the third plane 
contains the corresponding UV map. 

What format is the depth map, and confident map. Is it 16 bit unsigned? I've seen examples where floats are extracted from the depth stream, so I'm a bit confused. 

 

 

Leave a Comment

Please sign in to add a comment. Not a member? Join today