Intel® Perceptual Computing SDK - How to Use the Face Detection Module

Intel® Perceptual Computing SDK

The Intel® Perceptual Computing SDK is a library of pattern detection and recognition algorithm implementations exposed through standardized interfaces. The SDK provides a suite of face analysis algorithms including face location detection, landmark detection, face recognition and face attribute detection.

This tutorial shows how to use the SDK for face location detection, landmark detection, and how to write an application for these face analysis modules.

General Description

As shown in Figure 1, the face detection algorithm locates the rectangle position of a face or multiple faces from an image or a video sequence in real-time capture or playback mode.

Figure 1: Face location detection, the number 100 indicates the face ID.

In Figure 2, the landmark detection algorithm locates the 6 point or 7 point landmarks namely, the outer and inner corners of the eyes, the tip of the noise, and the outer corners of the mouth.

Figure 2: Face landmark detection, 7 point landmarks.

Procedure for a Simple Face Analysis Application

It is easy to create a simple face analysis application that will read color video from the camera, landmark detect, track the faces in each frame, and render them.

The code uses the UtilPipeline utility class for a simplified asynchronous pipeline implementation and the FaceRender class for rendering. The application will first initialize the pipeline and enable landmark detection. Then it will process each frame using the UtilPipeline ::LoopFrames function. To observe the results, the application will render each frame with the detection and landmark data using the FaceRender utility.

When the application is running, you should see the color rendering window with the landmark points detected as seen in Figure 2. When you click on the close button on the top, the application closes.

The whole code is shown in Figure 3 below:

Figure 3: A simple landmark detection code.

Project Setup and Configuration

For project setup and configuration,  please follow the steps in the “GettingStarted.docx” tutorial that describe “Microsoft Visual Studio 2008/2010 Setup” to create and configure the project. After completing the project setup and configuration, add the $(PCSDK_DIR)/include to the Additional Include Directories in the C/C++ project settings, as shown in Figure 4 below.

Figure 4: Project setup

Lastly, add the files shown in Figure 5 to the project from the “sample\common\” and “sample\landmark_detection\” directories.

Figure 5: Files to add to the project

It will be easier to understand the code by examing the code in steps as follows.

Code Explanantion

The first step is to use the UtilPipeline class to build a very simple application and define a new class “FacePipeline” which inherits from this utility class. The FacePipeLine class is needed to render the face landmarks since this functionality is not a part of the UtilPipeline utility class. So the constructer declares a new FaceRender and the destructor cleans it.



#include "util_pipeline.h"

#include "face_render.h"

class FacePipeline: public UtilPipeline {

public:

    FacePipeline(void):UtilPipeline() {

        m_face_render = NULL;

 

        // Create a face renderer

 m_face_render = new  FaceRender(L"Face Viewer");

      

 // Enable the face landmark detector

 EnableFaceLandmark();

    }

 

       ~FacePipeline() {

              if (m_face_render != NULL) delete m_face_render;

       }

 

protected:

       FaceRender*         m_face_render;

};

In order to render the landmarks, the application will need to implement the UtilPipeline::OnNewFrame function that renders face analysis detection data using FaceRender utility. The OnNewFrame function uses UtilPipeline::QueryFace() to query the face analyzer class. Then the face analyzer uses it’s PXCFaceAnalysis:: QueryFace() function to query the detected face. If there is any face detected, then the applicaition must query the detection data or the landmark data after dynamically casting to the respective module PXCFaceAnalysis::Landmark. Finally, a call to  FaceRender::SetLandmarkData is used for setting the face analysis data that will be rendered when calling the FaceRender::RenderFrame routine.



    virtual bool OnNewFrame(void) {

 

        /* face */

        PXCFaceAnalysis *faceAnalyzer = QueryFace();

        PXCFaceAnalysis::Landmark *landmark =

           faceAnalyzer->DynamicCast<PXCFaceAnalysis::Landmark>();

 

        // loop all faces

        m_face_render->ClearData();

        for (int fidx = 0; ; fidx++) {

            pxcUID fid = 0;

            pxcU64 timeStamp = 0;

            pxcStatus sts = faceAnalyzer->QueryFace(fidx, &fid, &timeStamp);

            if (sts < PXC_STATUS_NO_ERROR) break; // no more faces

            m_face_render->SetLandmarkData (landmark, fid);

        }

        return(m_face_render->RenderFrame( QueryImage(PXCImage::IMAGE_TYPE_COLOR)) );

    }

To process each frame of the captured video sequence, the main function simply calls LoopFrames() function to capture and process each frame. The LoopFrames function also calls the OnNewFrame function for retrieving and rendering the detection data.



    FacePipeline* pipeline = new FacePipeline();

    pipeline->LoopFrames();

    delete pipeline;

A more detailed example of the PxcFaceAnalysis module without utilizing the UtilPipeline class is given in the following section.

Procedure for Face Analysis

The application uses the following procedure for the face detection operations on a still image or a video sequence:

1st Step

The SDK session is the very first object the face detection application must create to hold all I/O or algorithm modules. This is created by calling the PXCSession_Create function. Then the application uses the session functions to create a module instance of the face detection algorithms, which is a part of the PXCFaceAnalysis interface. So, the application first creates an instance of the face analysis interface PXCFaceAnalysis by calling the PXCSession::CreateImpl function with the interface identifier PXCFaceAnalysis::CUID.



    // Create a session

	    PXCSession *session;

	    PXCSession_Create(&session);

 

    // Create Face analyzer interface

	    PXCFaceAnalysis  *faceAnalyzer;

	    session->CreateImpl(PXCFaceAnalysis::CUID, (void**)&faceAnalyzer);

2nd Step

The second step is to configure the PXCFaceAnalysis interface. The application enumerates supported configurations by using the QueryProfile function. Each profile describes a supported configuration. The application sets the desired configuration parameters by using the SetProfile function.



    // Configure the face analysis interface

	    PXCFaceAnalysis::ProfileInfo faInfo;

	    faceAnalyzer->QueryProfile(0, &faInfo);

	    faceAnalyzer->SetProfile(&faInfo);

Profile ‘0’ is the first profile and is usually the default profile of the interface. The PXCFaceAnalysis::ProfileInfo describes the video streams capture info and general parameters that are common to the different face analysis modules (detection, landmark, etc.). Thus, after QueryProfile, the capture device and video streams can be configured for face analysis procedures according to the enumerated PXCFaceAnalysis::ProfileInfo. Therefore, the PXCFaceAnalysis interface and capture device configurations need to be done jointly as follows:



// Configure the face analysis interface

	PXCFaceAnalysis::ProfileInfo faInfo;

	faceAnalyzer->QueryProfile(0, &faInfo);

 

// Find capture device

	UtilCaptureFile capture(session,0,0);

	capture.LocateStreams(&faInfo.inputs);

 

// Set the face analysis interface

	faceAnalyzer->SetProfile(&faInfo);

Here the UtilCaptureFile tool is utilized for a simplified approach of using the capture devices and setting the appropriate capturing stream.

3rd Step

At this point, the application may selectively configure a few or all of the analysis modules, such as face location detection, face landmark detection, and other implemented face analysis modules. The procedure of configuring these modules is similar. As an example, for configuring the face location detection, the application derives the detection interface from the PXCFaceAnalysis interface by using the DynamicCast function, and then calls the QueryProfile and SetProfile functions.



// Configure the face detector interface

	PXCFaceAnalysis::Detection *faceDetector;

	faceDetector=faceAnalyzer->DynamicCast<PXCFaceAnalysis::Detection>();

 

// Set the face detector profile

	PXCFaceAnalysis::Detection::ProfileInfo dInfo={0};

	faceDetector->QueryProfile(0, &dInfo);

	faceDetector->SetProfile(&dInfo);

The PXCFaceAnalysis::Detection::ProfileInfo describes the supported view angle of the detection algorithm. The default ViewAngle is set to VIEW_ANGLE_MULTI, which is equivalent to the bit-OR’ed value of view angle 0 degrees, 45 degrees, frontal (facing the camera), 135 degrees, and 180 degrees. Similarly, the same steps are followed for configuring the face landmark detection module.



// Configure the face detector interface

	PXCFaceAnalysis::Landmark *landmarkDetector;

	landmarkDetector=faceAnalyzer->DynamicCast<PXCFaceAnalysis::Landmark>();

 

// Set the face landmark detector profile

	PXCFaceAnalysis::Landmark::ProfileInfo lInfo={0};

	landmarkDetector->QueryProfile(1, &lInfo);

	landmarkDetector->SetProfile(&lInfo);

To set the 6 or 7 point landmark detection QueryProfile 0 or 1, respectively.

4th Step

The application performs face analysis by calling the ProcessImageAsync function. If the face analysis is on a still image, the application provides the image at the input. If the face analysis is on a video sequence, the application provides one video frame at a time in a main loop. The ProcessImageAsync function returns a SP, which the application can synchronize with before retrieving the analysis results.



PXCSmartArray<PXCImage> images; // declare image interface

	PXCSmartSPArray sps(2); //declare Sync points for asynchronous execution

 

// Read frame

	pxcStatus sts = capture.ReadStreamAsync(images,&sps[0]);

 

// Process frame using face detection

	sts = faceAnalyzer->ProcessImageAsync(images, &sps[1]);

	if (sts<PXC_STATUS_NO_ERROR) break;

 

// Synchronize both sync points

	sts = sps.SynchronizeEx();

	if (sps[0]->Synchronize(0)<PXC_STATUS_NO_ERROR) break; // test for EOF

For each captured frame, an asynchronous pipeline is executed that consists of reading a frame and then processing the current frame using any of the configured face analysis algorithms. Note that nothing is processed in the asynchronous pipeline until the Sync Points are synchronized by calling the SynchronizeEx function. The Asynchronous pipeline for Face Landmark detection is shown in Figure 6.

Figure 6: face landmark detection pipeline

5th Step

To retrieve the face detection results, the application needs to use QueryData function in the Detection interface. QueryFace is first called to enumerate all the detected faces during face analysis processing.



for (int fidx = 0; ; fidx++) {

	pxcUID fid = 0; pxcU64 timeStamp = 0;

	       sts = faceAnalyzer->QueryFace(fidx, &fid, &timeStamp);

	       if (sts < PXC_STATUS_NO_ERROR) break; // no more faces detected

       PXCFaceAnalysis::Detection::Data face_data;

	       faceDetector->QueryData(fid, &face_data);

	}

To retrieve the face landmark detection results, the application needs to QueryData for the labels in the already set profile. As follows:



for (int fidx = 0; ; fidx++) {

	    pxcUID fid = 0; pxcU64 timeStamp = 0;

	    sts = faceAnalyzer->QueryFace(fidx, &fid, &timeStamp);

	    if (sts < PXC_STATUS_NO_ERROR) break; // no more faces detected

 

    PXCFaceAnalysis::Landmark::ProfileInfo lInfo={0};

    landmark->QueryProfile(&lInfo);

 

    PXCFaceAnalysis::Landmark::LandmarkData data[7];

    pxcStatus sts=landmark->QueryLandmarkData(fid,lInfo.labels,&data[0]);

	}

Exercise: Putting it all together

After explaining key parts of the application above, run the landmark detection sample code shown in the boxes below. Copy the code and construct a project following the section “Project Setup and Configuration” explained previously in this tutorial. Similarly, add the following files shown in Figure 7 to the project.

Figure 7: Project files to be added

 



#include "pxcsession.h"

#include "pxccapture.h"

#include "pxcsmartptr.h"

#include "face_render.h"

#include "util_capture_file.h"

#include "util_cmdline.h"

#include "pxcface.h"

 

int wmain(int argc, wchar_t* argv[])

{  

    // Create a session

    PXCSmartPtr<PXCSession> session;

    pxcStatus sts=PXCSession_Create(&session);

       if (sts<PXC_STATUS_NO_ERROR) {

              wprintf_s(L"Failed to create the SDK sessionn");

              return 3;

       }

 

    UtilCmdLine cmdl(session);

    if (!cmdl.Parse(L"-sdname-nframes-file-record",argc,argv)) return 3;

 

    // Init Face analyzer

    PXCSmartPtr<PXCFaceAnalysis> faceAnalyzer;

    sts=session->CreateImpl(cmdl.m_iuid, PXCFaceAnalysis::CUID, (void**)&faceAnalyzer);

       if (sts<PXC_STATUS_NO_ERROR) {

              wprintf_s(L"Failed to locate a face module");

              return 3;

       }

 

       // Retrieve the input requirements

    PXCFaceAnalysis::ProfileInfo faInfo;

    faceAnalyzer->QueryProfile(0, &faInfo);

   

    // Find capture device

    UtilCaptureFile capture(session,cmdl.m_recordedFile,cmdl.m_bRecord);

    if (cmdl.m_sdname) capture.SetFilter(cmdl.m_sdname); /*L"Integrated Camera"*/

    sts=capture.LocateStreams(&faInfo.inputs);

       if (sts<PXC_STATUS_NO_ERROR) {

              wprintf_s(L"Failed to locate an input device that matches input for face analysisn");

              return 3;

       }

    faceAnalyzer->SetProfile(&faInfo);

 

    // Create detector instance

    PXCFaceAnalysis::Detection *faceDetector=faceAnalyzer->DynamicCast<PXCFaceAnalysis::Detection>();

    if (!faceDetector) {

              wprintf_s(L"Failed to locate the face detection functionalityn");

              return 3;

       }

 

 



    // Create landmark instance

    PXCFaceAnalysis::Landmark *landmarkDetector=landmarkDetector=faceAnalyzer->DynamicCast<PXCFaceAnalysis::Landmark>();

       if (!landmarkDetector) {

              wprintf_s(L"Failed to locate the face landmark functionalityn");

              return 3;

       }

 

    // Set landmark profile

    PXCFaceAnalysis::Landmark::ProfileInfo lInfo={0};

    landmarkDetector->QueryProfile(0, &lInfo);

    landmarkDetector->SetProfile(&lInfo);

 

    // Create Renderer

    PXCSmartPtr<FaceRender> faceRender(new FaceRender(L"Landmark Detection Sample"));

 

    int fnum;

    for (fnum=0;fnum<cmdl.m_nframes;fnum++) {

        PXCSmartArray<PXCImage> images;

        PXCSmartSPArray sps(2);

 

        ///* read and process frame */

        sts = capture.ReadStreamAsync(images,&sps[0]);

        if (sts<PXC_STATUS_NO_ERROR) break; // EOF

       

        sts = faceAnalyzer->ProcessImageAsync(images, &sps[1]);

        sts = sps.SynchronizeEx();

        if (sps[0]->Synchronize(0)<PXC_STATUS_NO_ERROR) break; // EOF

      

        // loop all faces

        faceRender->ClearData();

        for (int fidx = 0; ; fidx++) {

            pxcUID fid = 0;

            pxcU64 timeStamp = 0;

            sts = faceAnalyzer->QueryFace(fidx, &fid, &timeStamp);

            if (sts < PXC_STATUS_NO_ERROR) break; // no more faces

 

            // Query face detection results

            PXCFaceAnalysis::Detection::Data face_data;

            faceDetector->QueryData(fid, &face_data);

            faceRender->SetDetectionData(&face_data);

 

            // Query landmark points

            faceRender->SetLandmarkData(landmarkDetector, fid);

            faceRender->PrintLandmarkData(landmarkDetector, fid);

            wprintf_s(L"timestamp=%I64d, frame=%dn", timeStamp, fnum);

        }

 

        if (!faceRender->RenderFrame(images[0])) break;

    }

       return 0;

}

Running the application will output the face detection window and landmarks shown in Figure 8. Note that the FaceRender utility is used to render the output frame with the face detection results. The functions FaceRender::SetDetectionData and FaceRender::SetLandmarkData are used to set the face detection/landmark data and the function FaceRender::RenderFrame is used to render each frame with the set face detection data.

Figure 8: Landmark detection results

Per informazioni più dettagliate sulle ottimizzazioni basate su compilatore, vedere il nostro Avviso sull'ottimizzazione.
Contrassegni: