How to generate a real time 3D (mesh) model in Unity using Kinect sensor.

Greetings,

I’m currently developing an application with the initial goal of obtaining, in real time, a 3D model of the environment “seen” by a Kinect device. This information would be later on used for projection mapping but that’s not an issue, for the moment.

There are a couple of challenges to overcome, namely the fact that the Kinect will be mounted on a mobile platform (robot) and the model generation has to be in real-time (or close to it).

After a long research on this topic, I came up with several possible (?) architectures:

  1. Use the depth data obtained from Kinect, convert it into a point cloud (using PCL for this step), then a Mesh and then export it into Unity for further work.

  2. Use the depth data obtained from Kinect, convert it into a point cloud (using PCL for this step), export it into Unity and then convert it into a Mesh.

  3. Use KinectFusion that already the option of creating a Mesh model, and (somehow) automatically load the Mesh model created into Unity.

  4. Use OpenNI+ZDK (+ wrapper) to obtain the depth map and generate the Mesh using Unity.

Quite honestly, I’m kinda lost here, my main issue is the real-time requirement along with being forced to integrate several software components makes this it tricky problem. I don’t know which if any of these solutions are viable and the information/tutorials on these issues isn’t exactly abundant like the one, for example, for Skeleton tracking .

Any sort of help would be greatly appreciated.

Regards,
Nuno

Fair enough, let me put this way: how to generate a dynamic mesh (in real-time or close to it) using Unity based on what the Kinect is currently seeing?

I didn’t want to just throw out this question alone since unfortunately the answer to this question will most likely have to include the use of other frameworks/libraries (mentioned in my original post).

There are already wrappers using OpenNI (ZigFU) or Microsoft Kinect SDK which give access to Kinects depth information within Unity but is that alone enough to build a Mesh using only Unity?