Getting the color of a particular pixel in the depth image is not so tough because we have
Runtime.NuiCamera.GetColorPixelCoordinatesFromDepthPixel(). But if we are using the color image to do some kind of detection, how do we get the depth of any of the pixels? There doesn’t seem to be anything build into the Kinect SDK Beta, but I have managed to accomplish this through reversing the above method.
Here is the code snippet.
public static Point depthToColor;
public static Point colorToDepth;
private static void InitializeDepthToColorMapping()
depthToColor = new Point[DEPTH_IMAGE_WIDTH];
colorToDepth = new Point[COLOR_IMAGE_WIDTH];
int colorX, colorY;
for (colorX = 0; colorX < COLOR_IMAGE_WIDTH; colorX++)
// initialise rows of the colorToDepth matrix
colorToDepth[colorX] = new Point[COLOR_IMAGE_HEIGHT];
for (int depthX = 0; depthX < DEPTH_IMAGE_WIDTH; depthX++)
// initialise row
depthToColor[depthX] = new Point[DEPTH_IMAGE_WIDTH];
for (int depthY = 0; depthY = 0 && colorY >= 0 )
if (colorX < COLOR_IMAGE_WIDTH && colorY < COLOR_IMAGE_HEIGHT)
colorToDepth[colorX][colorY] = new Point(depthX, depthY);
if (colorX + 1 < COLOR_IMAGE_WIDTH)
colorToDepth[colorX + 1][colorY] = new Point(depthX, depthY);
if (colorY + 1 < COLOR_IMAGE_HEIGHT)
colorToDepth[colorX][colorY + 1] = new Point(depthX, depthY);
if (colorX + 1 < COLOR_IMAGE_WIDTH
&& colorY + 1 < COLOR_IMAGE_HEIGHT)
colorToDepth[colorX + 1][colorY + 1] =
new Point(depthX, depthY);
In this code,
nui is the name of the
Runtime instance. I created two 2-dimensional arrays,
colorToDepth. The size of each depends on the size of the image that is being used. For example, the depth image is 320×240, so depthToColor is a 320×240 array. Likewise,
colorToDepth is 640×480. The indexes correspond to the x and y of the pixel in the image, so
colorToDepth returns the coordinates of a pixel in the depth image corresponding with a pixel in the color image located at (321, 123).
There’s a bit of a trick due to the depth image has a smaller viewing area than the color image. That means any pixels around the top, left, or right borders of the color image have no depth information. Therefore, we leave those values to (0, 0). We should also set the depth of (0, 0) to 0, which means that no proper depth has been computed. If something is too close to the camera, too far away, or is an IR shadow zone, then the depth value will be 0.
Finally the way the algorithm works is to fill every the
depthToColor array using
GetColorPixelCoordinatesFromDepthPixel(). This is fine, because there approximately four color pixels corresponding to each depth pixel, so there is a direct correspondence. However, the next step is to reverse the mapping to
colorToDepth. In this case, there aren’t actually enough unique depth pixels to give each color pixel a unique depth. That’s no problem. I just assign the same depth to each of the four pixels in a square.
Future work could aim to make the interpolation better for
colorToDepth. Since there’s not exactly 4 color pixels per depth pixel, the algorithm above will lose a small amount of depth variance. This could be done with a median shell filter.