Introduction to Intel Perceptual Computing SDK: HelloCam!

Download


Introduction to Intel® Perceptual Computing SDK: HelloCam [PDF 305 KB]

When you want to learn a new programming language or SDK, the first step is usually is to write a program called "Hello World!" Here, we are talking about how to use the web cam features of the Intel® Perceptual Computing SDK, so we will call our first program "Hello Cam!"

But first, a bit of architecture. The following figure shows the architectural model of the Intel Perceptual Computing SDK:


Figure 1. Intel Perceptual Computing SDK Architecture

In this article we will build an example using the functionality provided by .NET porting of Perceptual SDK (shown in red in Figure 1).
In particular, we will use the classes inside libpxcclr.dll, which is located in the folders:

$(PCSDK_DIR)binwin32
$(PCSDK_DIR)binx64 

This library is a wrapper for the C++ classes and functionalities by exposing them to the .NET world (both C# and VB.NET).

The Windows Form project


To begin, we open Visual Studio* 2012 and create a new Windows Forms project as shown in the following figure.

Once the project is created, we need to add a reference to the dll using the command "Add Reference ..."

In this project, we will use the features made available by the class UtilMPipeline. This class works as a bridge between the .NET world and class UtilPipeline (written in C++ and documented here http://software.intel.com/sites/landingpage/perceptual_computing/documentation/html/utilpipeline.html) . This class is the easiest way to access the functionalities offered by the SDK.

In this sample, we will just acquire images from the web cam and show them in our GUI. In particular, we implement our own class that extends the UtilMPipeline.

Public Class ImageCamPipeline
    Inherits UtilMPipeline
    Public Sub New()
        MyBase.New()
        EnableImage(PXCMImage.ColorFormat.COLOR_FORMAT_RGB24, 640, 480)
    End Sub

End Class

The class constructor enables us to use the web cam module of the base class (in our example the image format is RGB 24 bit and the size is 640 by 480 pixels) called the EnableImage method. We will see later the other possibilities offered by EnableImage method and by the enumeration PXCMImage.ColorFormat.

First of all, we can note that all "managed" classes (.NET world) have the letter M in between the prefix that indicates the area of use and a suffix that indicates the functionalities. For example, the PXCMImage is the .NET class that implements the interface PXCImage of the Perceptual Computing framework.

To manage the image flow from the web cam and show them to our GUI, we need to override the OnImage method of UtilMPipeline.

Public Overrides Sub OnImage(image As PXCMImage)
    Dim data As PXCMImage.ImageData
    Dim status = image.AcquireAccess(PXCMImage.Access.ACCESS_READ,
                                     PXCMImage.ColorFormat.COLOR_FORMAT_RGB32, data)
    If status = pxcmStatus.PXCM_STATUS_NO_ERROR Then
        Dim width As UInt32
        Dim height As UInt32
        Dim colorFormat As PXCMImage.ColorFormat
        If image.GetImageInfo(width, height, colorFormat) Then
            Dim bitmap As Bitmap = data.ToBitmap(width, height)
            RaiseEvent ImageCaptured(Me, New ImageCapturedEventArgs() With {.Image = bitmap,
                                                                            .TimeStamp = DateTime.Now,
                                                                            .FPS = CalculateFPS(DateTime.Now)})
            image.ReleaseAccess(data)
        End If
    End If
End Sub
 

ImageCamPipeline fires the ImageCaptured event when a new image is ready and the GUI can manage the event to show the image.

Public Event ImageCaptured(sender As Object, e As ImageCapturedEventArgs)

In our sample, the event argument contains the image, the capture timestamp (date and time) and the instantaneous frame rate (calculated as the difference between the timestamps of two consecutive images).

Public Class ImageCapturedEventArgs
    Inherits EventArgs
    Public Property Image As Bitmap
    Public Property TimeStamp As DateTime
    Public Property FPS As Integer?
End Class




The instantaneous frame rate is calculated from the difference of timestamps of two consecutive images using the following function:
    Private Function CalculateFPS(currentTimeStamp As Date) As Integer?
        Dim fps As Integer? = Nothing
        If LastImageTimestamp.HasValue Then
            fps = CInt(Math.Floor(1000D / (currentTimeStamp - LastImageTimestamp.Value).TotalMilliseconds))
        End If
        Me.LastImageTimestamp = currentTimeStamp
        Return fps
    End Function



Once we have created the class that interacts with the SDK functionality, we can begin to create the interface that will display the images and the frame rate.
The following figure shows the Visual Studio designer for our form:


The code behind of the form is quite simple:

Public Class Form1
    Private pipeline As ImageCamPipeline
   Private Sub Form1_FormClosing(sender As Object, e As FormClosingEventArgs) Handles Me.FormClosing
        pipeline.Dispose()
    End Sub
    Private Sub Form1_Load(sender As Object, e As EventArgs) Handles Me.Load
        pipeline = New ImageCamPipeline
        AddHandler pipeline.ImageCaptured, AddressOf ImageCapturedHandler
        pipeline.LoopFrames()
    End Sub
    Private Sub ImageCapturedHandler(sender As Object, e As ImageCapturedEventArgs)
        UpdateInterface(e)
    End Sub
    Private Delegate Sub UpdateInterfaceDelegate(e As ImageCapturedEventArgs)
    Public Sub UpdateInterface(e As ImageCapturedEventArgs)
        If Me.InvokeRequired Then
            Me.Invoke(New UpdateInterfaceDelegate(AddressOf UpdateInterface), e)
        Else
            pctImage.Image = e.Image
            If e.FPS.HasValue Then
                lblFPS.Text = String.Format("{0} fps", e.FPS.Value)
            Else
                lblFPS.Text = String.Empty
            End If
        End If
    End Sub
End Class

The form has an instance of ImageCamPipeline that is created in the load form event. Once we have created the ImageCamPipeline's instance, we add the event handler for ImageCaptured and finally we call the LoopFrames method to start the image capture.

The LoopFrames method, as we can read in the official documentation (http://software.intel.com/sites/landingpage/perceptual_computing/documentation/html/loopframes_util pipeline.html), starts a loop that acquires and releases frames. Reading the official documentation, we see that the pseudo-code executed by the method is as follows:

if (!Init()) return false;
 for (;;) {
         // some device hot-plug code omitted.
         if (!AcquireFrame(true)) break;
         if (!ReleaseFrame()) break;
 }
 Close();
 return true

This class performs three main steps. It:

  1. is initialized (Init method)
  2. runs an endless loop in which the frame is retrieved from the cam and, subsequently, released (AcquireFrame and ReleaseFrame methods)
  3. closes the connection with the device (Close method)

LoopFrames blocks the thread that calls it, and in our interface, this means that the GUI is unable to start because the LoopFrames method is called in the Load event handler that takes place in UI main thread.
To solve the problem, we can override the LoopFrames method so that the LoopFrames method of the base class is run in a new thread. The new thread won't block the main thread and our UI can start.

    Public Overrides Function LoopFrames() As Boolean
        If LoopFramesThread Is Nothing Then
            LoopFramesThread = New Thread(New ThreadStart(AddressOf LoopFramesCode))
            LoopFramesThread.Start()
            Return True
        Else
            Return False
        End If
    End Function
    Private LoopFramesThread As Thread
     Private Sub LoopFramesCode()
        Try
            RaiseEvent PipelineStarted(Me, EventArgs.Empty)
            Dim result = MyBase.LoopFrames()
            If Not result Then RaiseEvent PipelineError(Me, EventArgs.Empty)
        Catch ex As Exception
            RaiseEvent PipelineStopped(Me, EventArgs.Empty)
        End Try
    End Sub

We use Try...Catch in the method to manage the thread abort that may occur when we close the application.
The PipelineStarted, PipelineError e PipelineStopped events allow us to notify the UI when the pipeline starts if an error occurs and the pipeline is stopped.
We might have an error if we try to use a feature not available on the web cam that we are using. For example, if we try to take a depth image and we are not using a Creative* Interactive Gesture Camera.

Redefining the LoopFrames method, our pipeline is multi-threaded and the GUI is fluid and responsive. As the LoopFrames method is performed in a secondary thread, the ImageCaptured event, raised from the pipeline, occurs in a thread that is not the UI main thread.

Since Windows Forms are STAs (single-threaded apartments), we are forced to update our interface using the InvokeRequired property plus the Invoke method (UpdateInterface method). The sample is quite simple, but it allows you to see how the Perceptual Computing SDK works.
One of the interesting features of the Perceptual Computing SDK is that certain features will work even if you are not using a Creative Cam. For example, if you retrieve RGB images from the web cam, you can run the code also using your PC’s web cam.

With a Creative Cam you can retrieve other types of images (such as Depth image) and, in general, increase the performance. For example, the following figure shows a comparison between the integrated web cam on my Ultrabook™ system and Creative Cam in terms of definition and, especially, of instantaneous frame rate.


On a Creative Cam, we can retrieve the Depth image simply by changing the pipeline constructor:

    Public Sub New()
        MyBase.New()
        EnableImage(PXCMImage.ColorFormat.COLOR_FORMAT_DEPTH, 320, 240)
    End Sub

The following figure shows the depth image:

The set of values for the PCXMImage.ColorFormat enumeration is available at http://software.intel.com/sites/landingpage/perceptual_computing/documentation/html/index.html?pxcimagecolorformat.html.

 

Intel, the Intel logo, and Ultrabook are trademarks of Intel Corporation in the U.S. and/or other countries.
Copyright © 2013 Intel Corporation. All rights reserved.
*Other names and brands may be claimed as the property of others.

Para obtener más información sobre las optimizaciones del compilador, consulte el aviso sobre la optimización.