Integrating WPF and Microsoft Kinect SDK with OpenTK

Well, this is my first post, so I'm gonna make it a simple (yet not so short) one.

If you have started messing around with OpenTK recently and also do not have much experience with .NET, this might help you. This is especially true if you want to use WPF (Windows Presentation Foundation) instead of Windows Forms for rendering OpenGL graphics through OpenTK. And if, on top of that, you also intend on using Microsoft Kinect SDK, then this is the right post.


Using Kinect and OpenGL simultaneously gives us a really wide range of possibilities, for instance real time rendering of the 3D space captured by Kinect (plus texture). One way of obtaining that is having access to both Kinect and OpenTK's data and events, in the same application.

For this post, I will just focus on how to have a simple application that can render OpenTK graphics and present video from Kinect, in the same window.

Well, let's start then:

1. Create a new WPF Application project and rename the MainWindow Grid to grid.

2. Add references in the project to:

• System.Windows.Forms
• WindowsFormsIntegration
• OpenTK
• OpenTK.GLControl
• System.Drawing (to ease the use of colors in OpenTK)
• Microsoft.Research.Kinect (to use the Kinect device)

3. Open the WPF Designer and add the following namespace mapping (in the source code area) to the Window element:
xmlns:wf="clr-namespace:System.Windows.Forms;assembly=System.Windows.Forms"

Then, add an Image element (just to show Kinect video stream) called image and a WindowsFormsHost element called host, both inside the Grid element, side by side. Set an event for when the Window element is Loaded. It should look something like this:

<Window x:Class="WpfApplication1.MainWindow"
        xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
        xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
        xmlns:wf="clr-namespace:System.Windows.Forms;assembly=System.Windows.Forms"
        Title="MainWindow" Height="375" Width="1000" Loaded="Window_Loaded">
    <Grid Name="grid">
        <Image Width="500" Height="375" Name="image" HorizontalAlignment="Left" />
        <WindowsFormsHost Width="500" Height="375" Name="host" HorizontalAlignment="Right" />
    </Grid>
</Window>

4. In the .cs include the following types:

using System.Windows.Forms;
using OpenTK;
using OpenTK.Graphics;
using OpenTK.Graphics.OpenGL;
using Microsoft.Research.Kinect.Nui;

5. Make the Window element Loaded event handler look like this:

        private void Window_Loaded(object sender, RoutedEventArgs e)
        {
            // Create the GLControl.
            glc = new GLControl();

            // Assign the GLControl as the host control's child.
            host.Child = glc;
        }

You are now creating an instance of GLControl and associating it with host (the WindowsFormsHost element you added) by making glc its child.

6. Next is assigning events Load and Paint of GLControl to two new event handlers (you can double press Tab key to automatically create an event handler body, after the += sign):

        private void Window_Loaded(object sender, RoutedEventArgs e)
        {
            // Create the GLControl.
            glc = new GLControl();

            // Assign Load and Paint events of GLControl.
            glc.Load += new EventHandler(glc_Load);
            glc.Paint += new System.Windows.Forms.PaintEventHandler(glc_Paint);

            // Assign the GLControl as the host control's child.
            host.Child = glc;
        }

        void glc_Paint(object sender, System.Windows.Forms.PaintEventArgs e)
        {
            throw new NotImplementedException();
        }

        void glc_Load(object sender, EventArgs e)
        {
            throw new NotImplementedException();
        }

7. Now fill Load and Paint event handlers with an initial OpenGL configuration (viewport, etc) and what should be drawn every time the control needs a repaint, respectively. The result can be something like:

        void glc_Load(object sender, EventArgs e)
        {
            // Make background "chocolate"
            GL.ClearColor(System.Drawing.Color.Chocolate);

            int w = glc.Width;
            int h = glc.Height;

            // Set up initial modes
            GL.MatrixMode(MatrixMode.Projection);
            GL.LoadIdentity();
            GL.Ortho(0, w, 0, h, -1, 1);
            GL.Viewport(0, 0, w, h);
        }

        void glc_Paint(object sender, System.Windows.Forms.PaintEventArgs e)
        {
            GL.Clear(ClearBufferMask.ColorBufferBit | ClearBufferMask.DepthBufferBit);

            GL.MatrixMode(MatrixMode.Modelview);
            GL.LoadIdentity();

            // Draw a little yellow triangle
            GL.Color3(System.Drawing.Color.Yellow);
            GL.Begin(BeginMode.Triangles);
            GL.Vertex2(200, 50);
            GL.Vertex2(200, 200);
            GL.Vertex2(100, 50);
            GL.End();

            glc.SwapBuffers();
        }

Build and run it. If you (or me) haven't forgot something, it should be working and showing a little yellow triangle inside the WPF Grid.

8. Now, we want to have access to Kinect data, in real time.
You added a reference to Microsoft.Research.Kinect and did using Microsoft.Research.Kinect.Nui; in the beginning of this post, like you'd do with a normal Kinect project (it's exactly the same, now that we have finished integrating OpenTK), so now it's time to get Kinect ready.
Create a Runtime instance in the class definition, initiate it and assign the required events in the Window Loaded event.

            // Initiate Kinect runtime and streams
            nui = Runtime.Kinects[0];
            try
            {
                nui.Initialize(RuntimeOptions.UseColor);
                nui.VideoStream.Open(ImageStreamType.Video, 2,
                    ImageResolution.Resolution640x480, ImageType.Color);
            }
            catch (InvalidOperationException)
            {
                return;
            }
            // Assign stream events
            nui.VideoFrameReady += new EventHandler(nui_VideoFrameReady);

9. Fill the video stream ready event handler to just dump the stream to the Image element we previously created.

        public void nui_VideoFrameReady(object sender, ImageFrameReadyEventArgs e)
        {
            PlanarImage Image = e.ImageFrame.Image;
            image.Source = BitmapSource.Create(Image.Width, Image.Height, 96, 96,
                PixelFormats.Bgr32, null, Image.Bits, Image.Width * Image.BytesPerPixel);
        }

Build and run, hopefully you'll see a window showing the Kinect video stream and an OpenGL rendering via OpenTK, both throwing and handling whatever events you register to them.

Full source code: Download here.

Everything I explained here has already been done on other sources. I've just put things together for a simple way of integrating Kinect (and WPF) with OpenTK (OpenGL in .NET). All source code has been tested on OpenTK 1.0 and Microsoft Kinect SDK beta 2.

For more information, check the following references:
Building a Windows.Forms + GLControl based application
Hosting a Windows Forms Control in WPF by Using XAML
Add or Remove References in Visual Studio