rues
s
is Up
nDown s
ur
bpqd ownHome

CHIRPYERBIRDY CAN
(startup wantnARYd own)whipp
ipp
ipp
ed
d
ippN undipp

T-TUB SMITHY

dance(
-ing-
ayin
in)!
ed (t s-
aid she s-

PLUS UPAYMORE REDOS

aysomesing
keep)i?
re+als
(you w
ill(submit que)ry ubs
tances)
elete indeed
HomE

QUOTE RAREST OF OUR RUSES UNQUOTE

I have been working on a robot simulator. I just added control via Kinect.

Kinect-controlled robot simulator

First of all, the most helpful place to go for your questions about the Kinect SDK is the Microsoft forums:

http://social.msdn.microsoft.com/Forums/en-US/category/kinectsdk

A great application that probably does exactly what you want to do — transform the person into the Incredible Hulk while maintaining the background.

http://www.codeproject.com/KB/dotnet/KinectGettingStarted.aspx

So don’t you think it would be great to stream data from the Kinect to the Windows Phone 7. Basically, this is already what I’ve been trying to set up with Codeplex: Kinect Server. We want to get post-processed data from our Kinects into some other application. For example, we might send it to Blender or Autodesk Softimage for making animations. We might send it to Garry’s Mod to do some cool game world.

And now they are going to show us exactly how we might set up an optimized server to stream data to the Windows Phone 7.

The text from http://create.msdn.com/en-us/education/talks currently says:

October Kinect and Windows Phone Playing Together

In this session we will go through creating a Kinect-based Socket Server that will allow clients to connect and download real time skeletal data. The client we will build in this session will be a Windows Phone application based upon the Mango release to leverage the new sockets support.

Date: October 18, 2011 from 9:00AM-10:00AM PST

Register today.

I have already registered for this. I put it into time converter and it shows this:

Well, all that means is that I’ll need to catch it at 2:30 am. If you are dedicated like me you will catch the live event. If you are reading this and the event has already gone past, you should be able to watch to recorded video. Good luck!

Getting the color of a particular pixel in the depth image is not so tough because we have Runtime.NuiCamera.GetColorPixelCoordinatesFromDepthPixel(). But if we are using the color image to do some kind of detection, how do we get the depth of any of the pixels? There doesn’t seem to be anything build into the Kinect SDK Beta, but I have managed to accomplish this through reversing the above method.

Here is the code snippet.


public static Point[][] depthToColor;
public static Point[][] colorToDepth;
private static void InitializeDepthToColorMapping()
{
    depthToColor = new Point[DEPTH_IMAGE_WIDTH][];
    colorToDepth = new Point[COLOR_IMAGE_WIDTH][];
    int colorX, colorY;
    for (colorX = 0; colorX < COLOR_IMAGE_WIDTH; colorX++)
    {
        // initialise rows of the colorToDepth matrix
        colorToDepth[colorX] = new Point[COLOR_IMAGE_HEIGHT];
    }
    for (int depthX = 0; depthX < DEPTH_IMAGE_WIDTH; depthX++)
    {
        // initialise row
        depthToColor[depthX] = new Point[DEPTH_IMAGE_WIDTH];
        for (int depthY = 0; depthY = 0 && colorY >= 0 )
        {
            if (colorX < COLOR_IMAGE_WIDTH && colorY < COLOR_IMAGE_HEIGHT)
            {
                colorToDepth[colorX][colorY] = new Point(depthX, depthY);
                if (colorX + 1 < COLOR_IMAGE_WIDTH)
                    colorToDepth[colorX + 1][colorY] = new Point(depthX, depthY);
                if (colorY + 1 < COLOR_IMAGE_HEIGHT)
                    colorToDepth[colorX][colorY + 1] = new Point(depthX, depthY);
                if (colorX + 1 < COLOR_IMAGE_WIDTH
                        && colorY + 1 < COLOR_IMAGE_HEIGHT)
                    colorToDepth[colorX + 1][colorY + 1] =
                        new Point(depthX, depthY);
                    }
                }
            }
            catch (Exception)
            {
                System.Windows.Forms.MessageBox.Show("ERROR");
            }
        }
    }           
}

In this code, nui is the name of the Runtime instance. I created two 2-dimensional arrays, depthToColor and colorToDepth. The size of each depends on the size of the image that is being used. For example, the depth image is 320×240, so depthToColor is a 320×240 array. Likewise, colorToDepth is 640×480. The indexes correspond to the x and y of the pixel in the image, so colorToDepth[321][123] returns the coordinates of a pixel in the depth image corresponding with a pixel in the color image located at (321, 123).

There’s a bit of a trick due to the depth image has a smaller viewing area than the color image. That means any pixels around the top, left, or right borders of the color image have no depth information. Therefore, we leave those values to (0, 0). We should also set the depth of (0, 0) to 0, which means that no proper depth has been computed. If something is too close to the camera, too far away, or is an IR shadow zone, then the depth value will be 0.

Finally the way the algorithm works is to fill every the depthToColor array using GetColorPixelCoordinatesFromDepthPixel(). This is fine, because there approximately four color pixels corresponding to each depth pixel, so there is a direct correspondence. However, the next step is to reverse the mapping to colorToDepth. In this case, there aren’t actually enough unique depth pixels to give each color pixel a unique depth. That’s no problem. I just assign the same depth to each of the four pixels in a square.

Future work could aim to make the interpolation better for colorToDepth. Since there’s not exactly 4 color pixels per depth pixel, the algorithm above will lose a small amount of depth variance. This could be done with a median shell filter.

There are a number of difficulties that people have with INUIInstanceHelper.dll and the Kinect SDK.  Just search google and you’ll find a lot of hits about these problems.

One of these is the error:

Could not load file or assembly 'INuiInstanceHelper, Version=1.0.0.10, Culture=neutral, PublicKeyToken=31bf3856ad364e35' or one of its dependencies. An attempt was made to load a program with an incorrect format.
This problem I found easy to fix by changing the build setting of the project that contains a reference to INuiInstanceHelper to “x86″. Remember, The Kinect SDK Beta only supports x86 projects. For the codeplex link, see http://stackoverflow.com/questions/6655555/inconsistent-issues-with-loading-inuiinstancehelper-in-kinect-sdk.

A related error is as follows:

Could not load file or assembly ‘INuiInstanceHelper.dll’ or one of it’s dependencies. The specified module could not be found.
This error seems to be related to the .dll not being installed correctly and requires using gacutil to fix it. Luckily I didn’t need to do that, since I don’t have admin privileges on my machine. But if that’s the problem, then you could try checking this post:

Soul Solutions Blog.

Getting started with the Microsoft Kinect SDK is pretty straightforward. Download it from Microsoft Research, install and you can test it by running the Sample Skeletal Viewer that is located in the Microsoft Kinect for Windows SDK BETA in the Start menu. You might also need the DirectX SDK to be installed. Finally, you want to download the Coding4Fun Kinect helper library for easy converting of images (http://c4fkinect.codeplex.com/ and click Download on the right side of the page). There’s a helpful video guide for a more in-depth installation walkthrough.

Start a new WPF C# application. Right click “References” in the Solution Explorer and add “Coding4Fun.Kinect.Wpf” and “Microsoft.Research.Kinect” via the Browse tab. If you have done this step at least once, you can use the Recent tab as well.

Now it comes to the application itself. Drag an image into the display space and call it image. Revise the MainWindow.xaml.cs code to look like this. You’ll get a window that shows the depth image in real time!

 
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Windows;
using System.Windows.Controls;
using System.Windows.Data;
using System.Windows.Documents;
using System.Windows.Input;
using System.Windows.Media;
using System.Windows.Media.Imaging;
using System.Windows.Navigation;
using System.Windows.Shapes;
using Tom_Anderson;
using Microsoft.Research.Kinect.Nui;
using Coding4Fun.Kinect.Wpf;
using System.Threading;

void nui_DepthFrameReady(object sender, ImageFrameReadyEventArgs e)
{
image.Source = ImageFrameExtensions.ToBitmapSource(e.ImageFrame);
}
}
}