Home › Articles

Downloads

Download White Paper: Using UMC De-multiplexer with the Intel® Media Software Development Kit [PDF 267KB]
Download Source Code: Source Zipfile [ZIP 9.57MB]

Abstract

To take advantage of present and future Intel graphics hardware acceleration and multi-core threading, the Intel® Media Software Development Kit (SDK) can be used to encode, transcode, decode video content. The decode engine in the Intel® Media SDK can decode a pure video stream (H.264/AVC, MPEG-2 video or VC-1).

However, given that video content usually resides in a container format, a de-multiplexer (or splitter) is needed to extract audio and video data from the media stream before passing the video data to the decode engine of the Intel® Media SDK. This paper leverages the existing Intel® Media SDK and Intel® IPP sample to describe how to use a de-multiplexer to handle decoding MP4/AVC streams.

Introduction

In this paper, we describe how to use the Unified Media Classes (UMC) de-multiplexer to extract the compressed video data. The compressed video data is then delivered to the decode engine in the Intel® Media SDK to produce uncompressed video frames. Although users can use the UMC decoder to accomplish this task, using the Intel® Media SDK is the preferred way since it is highly optimized and accelerated using the Intel platform media processing capabilities.

Figure 1 illustrates a possible video data processing scenario (transcode) using the UMC de-multiplexer / multiplexer (or splitter/muxer) and the Intel® Media SDK.


Figure 1. Usage Model of the Intel® Media SDK.

The MP4/AVC (or H.264/AVC [4]) container format is used as the input stream in this paper. This multimedia container format is commonly used to store digital video and audio streams. The corresponding UMC de-multiplexer that splits MP4/AVC streams is called UMC::MP4Splitter. This work is derived from the sample code named sample_decode shipped with the Intel Media SDK [1], and some de-multiplexer functionalities first developed in [3]. The result is a completely standalone sample code showing how we can integrate the UMC::MP4Splitter into the Intel Media SDK and decode a MP4/AVC media stream. The output video frames are written to a file in the NV12 format, or further processed by the video preprocessing and encode engines in SDK.

The Unified Media Classes

The Unified Media Classes (UMC) is a set of interfaces used to build standard blocks of media applications. UMC is part of the Intel® Integrated Performance Primitives [2] (Intel® IPP) samples and they are highly optimized for performance.

Users can get UMC by downloading the Intel® IPP samples. The UMC component is located in the folder /Samples/en_US/ipp-samples/audio-video-codecs.

UMC can be used in various decoding, encoding and transcoding applications. In particular, The UMC "Splitter Class" can be used for setting different splitters. In this paper we use the UMC::MP4Splitter, to split audio and video tracks from the input media streams. The MP4 splitter de-multiplexes a MP4/AVC bitstream, and insert start codes in the video bitstream (according to Annex B of the ITU-Rec. H.264 specification) This step is handled by the Frame Construction Class.

The following UMC::MP4Splitter class methods are used:

• Init: initializes the MP4 splitter with parameters in UMC::SplitterParams.
• GetInfo: Obtain information about the media stream.
• Run: starts the MP4 splitter.
• GetNextData: unlocks the previous frame and provides access to the next frame of a specific track.
• Stop: stops the splitter.
• Close: destroys all internal resources.

For details, refer to the Reference Guide "Intel Integrated Performance Primitives Unified Media Classes" [2].

The following sample code shows the splitter initialization. First, the input source file is validated in the file reader component. The splitter parameters are set. Based on the container information, the pointer to the video data is saved in pVideoDecSpecInfo for later use.

UMC::Status CDecodingPipeline::InitSplitter()
{
   UMC::Status umcRes = UMC::UMC_OK;
		
   umcRes = pReader->Init(&readerParams);

   if(UMC::UMC_OK == umcRes)
   {
	// Initialize splitter parameters
	pSplitParams->m_pDataReader		= pReader;
	pSplitParams->m_uiSelectedVideoPID	= 0; // ID for video stream chosen by user
	pSplitParams->m_lFlags			= UMC::VIDEO_SPLITTER;  // (Ipp32u) splitter's flags
	pSplitParams->m_pMemoryAllocator	= NULL;

	umcRes = pSplitter->Init(*pSplitParams);

	if(UMC::UMC_OK == umcRes)
	{
	   // Fetch meta data from container
	   umcRes = pSplitter->GetInfo(&pSplitterInfo);
	   if(UMC::UMC_OK == umcRes)
	   {
		// Fetch first video track from splitter
		umcRes = UMC::UMC_ERR_INVALID_STREAM;

		int numTracks = pSplitterInfo->m_nOfTracks;

		int i;
		for(i=0; i<numTracks; i++)
		{
		   if(pSplitterInfo->m_ppTrackInfo[i]->m_Type == UMC::TRACK_H264)
			break;
		}

		UMC::VideoStreamInfo *ptemp;
		ptemp = (UMC::VideoStreamInfo*)pSplitterInfo->m_ppTrackInfo[i]->m_pStreamInfo;

		if (ptemp->stream_type == UMC::H264_VIDEO)
		{
		   memcpy(&videoInfo,
			(UMC::VideoStreamInfo*)pSplitterInfo->m_ppTrackInfo[i]->m_pStreamInfo,
			sizeof(UMC::VideoStreamInfo));

		   pVideoDecSpecInfo = pSplitterInfo->m_ppTrackInfo[i]->m_pDecSpecInfo;
		   umcRes = UMC::UMC_OK;	
		}
	   }
	}
   }

   return umcRes;
}

The Frame Construction Class

With the above pointer to the video data, the method GetNextData in the UMC::MP4Splitter class can read video frames from a container format. Since these frames don't have start codes they need to be reformatted before sending to the decode engine.

To handle the additional steps required with this container format, the CFrameConstructor class is added. CFrameConstructor's reformats the video frames by inserting start codes. Please refer to the source code of CFrameConstructor for more details.

The mfxBitstream structure defines the buffer that holds compressed video data. The following sample code shows how the splitter can get frames and reformat them.

UMC::Status CDecodingPipeline::ReadFrames(mfxBitstream *pBS)
{
   UMC::MediaData	videoData;
   UMC::Status		umcRes = UMC::UMC_OK;

   mfxBitstream tempBS;

   if(!bEndOfStream && pBS->DataLength <= 4) 
   {
	pBS->DataOffset = 0;

	// Read ahead and put frames in Media SDK bitstream
	while(pBS->DataLength < BS_READ_BUFFER_SIZE && umcRes == UMC::UMC_OK)
	{
	   // fetch next frame from splitter
	   umcRes = pSplitter->GetNextData(&videoData, 0);

	   if(umcRes == UMC::UMC_OK)
	   {
		Frame++;

		// Reformat frame from splitter
		pFrameConstructor->ConstructFrame(&videoData, &tempBS);
		memcpy(pBS->Data + pBS->DataLength, tempBS.Data, tempBS.DataLength);
		pBS->DataLength += tempBS.DataLength;

		delete tempBS.Data;
	   }
	   else 
	   {
		// No more fram data in container
		bEndOfStream = true;
	   }
	}
   }

   return umcRes;
}

The Intel® Media SDK

The Intel® Media SDK is a software library that exposes the media acceleration capabilities of Intel platforms for decoding, encoding and video preprocessing [1].

There are three main categories of the Intel® Media SDK: the MFXVideoDECODE Class is responsible for decoding operations, the Video Preprocessing (VPP) MFXVideoVPP Class for video preprocessing operations, and the MFXVideoENCODE Class for encoding operations.

The MFXVideoDECODE Class takes compressed video bitstreams as input and decodes them to raw video frames. The VPP class takes raw video frames as input, performs preprocessing and generates raw video frames at output. The MFXVideoENCODE Class takes raw video frames as input, encodes and compresses them into the output bitstream.

The MFXVideoDECODE Class processes video bitstreams, but cannot process bitstream that reside in a container format. Therefore, the UMC splitter is used to de-mutiplex a MP4/AVC file format and then the output video bitstreams are sent to the MFXVideoDECODE Class.

The following methods in the MFXVideoDECODE Class are referred in the sample code:

• DecodeHeader: parses the input bitstream and fills the video parameters
• Init: allocates internal memory and validates the above video parameters
• QueryIOSurf: queries the number of frames required for decoding
• DecodeFrameAsync: decodes the input bitstream
• Close: terminates the decoding operations and de-allocates internal memory

The following sample code shows the initialization of the program. The member m_FileWriter, responsible for writing the frames to a file of format YUV, is initialized. Then an SDK session is initialized with the desired implementation (either hardware or software). The UMC::MP4Splitter is created, initialized and started processing the media stream (pSplitter->Run). The decoder is created with the corresponding video type. The optional input parameter pParams->bd3dAlloc tells the program to either allocate video frames in video memory (Direct3D* 9 surface) or in system memory. The structure m_mfxBS, which defines the compressed video buffer, is initialized. Video data from pVideoDecSpecInfo is reformatted and stored in m_mfxBS. The functions CreateAllocator and AllocFrames allocate the frames for the decoder. After that, the decoder can start.

mfxStatus CDecodingPipeline::Init(sInputParams *pParams)
{
    CHECK_POINTER(pParams, MFX_ERR_NULL_PTR);

    mfxStatus sts = MFX_ERR_NONE;
    UMC::Status umcRes = UMC::UMC_OK;

    // prepare YUV file writer
    sts = m_FileWriter.Init(pParams->strDstFile);
    CHECK_RESULT(sts, MFX_ERR_NONE, sts);  
  
    // init session
    mfxIMPL impl = pParams->bUseHWLib ? MFX_IMPL_HARDWARE : MFX_IMPL_SOFTWARE;
    mfxVersion version = {MFX_VERSION_MINOR, MFX_VERSION_MAJOR};     
    sts = m_mfxSession.Init(impl, &version);
    CHECK_RESULT(sts, MFX_ERR_NONE, sts);

    // Create MP4 splitter
    pSplitter			= new UMC::MP4Splitter();
    pSplitParams		= new UMC::SplitterParams();
    pFrameConstructor	= new CAVCFrameConstructor();

    umcRes = InitSplitter();

    // Kick-start splitter
    if (umcRes == UMC::UMC_OK)
	umcRes = pSplitter->Run();

    // create decoder
    m_pmfxDEC = new MFXVideoDECODE(m_mfxSession);    

    // CodecId supported
    m_mfxVideoParams.mfx.CodecId = MFX_CODEC_AVC;

    // set memory type
    m_bd3dAlloc = pParams->bd3dAlloc;

    // prepare bit stream
    sts = InitMfxBitstream(&m_mfxBS, 1024 * 1024);
    CHECK_RESULT(sts, MFX_ERR_NONE, sts);    

// --- Splitter integration ---
mfxBitstream tempBS;

pFrameConstructor->ConstructFrame(pVideoDecSpecInfo, &tempBS);
memcpy(m_mfxBS.Data, tempBS.Data, tempBS.DataLength);
m_mfxBS.DataLength = tempBS.DataLength;

delete tempBS.Data;

    sts = InitMfxParams();
    CHECK_RESULT(sts, MFX_ERR_NONE, sts);

    // init allocator 
sts = CreateAllocator();


    CHECK_RESULT(sts, MFX_ERR_NONE, sts);

// if allocator is provided to MediaSDK as external, frames must be allocated prior 
// to decoder initialization
    sts = AllocFrames();
    CHECK_RESULT(sts, MFX_ERR_NONE, sts);

    // init decoder
    sts = m_pmfxDEC->Init(&m_mfxVideoParams);
    IGNORE_MFX_STS(sts, MFX_WRN_PARTIAL_ACCELERATION);
    CHECK_RESULT(sts, MFX_ERR_NONE, sts);

    return MFX_ERR_NONE;
}

Running the decoder is shown in the following sample code. Basically, the decoder runs the following steps in a loop:

• waits if the hardware is busy
• reads more video data into a buffer
• obtains a free frame surface
• runs the decoding operation in the asynchronous mode
• synchronizes the result
• write any output video frames, if available, to a file

mfxStatus CDecodingPipeline::RunDecoding()
{   
    mfxSyncPoint        syncp;
    mfxFrameSurface1    *pmfxOutSurface = NULL;
    mfxStatus           sts = MFX_ERR_NONE;
    mfxU16              nIndex = 0; // index of free surface

    // print stream info 
    PrintInfo();    

    while (MFX_ERR_NONE <= sts || MFX_ERR_MORE_DATA == sts || MFX_ERR_MORE_SURFACE == sts)
    {
        if (MFX_WRN_DEVICE_BUSY == sts)
        {
            Sleep(5); // just wait and then repeat the same call to DecodeFrameAsync
        }
        else if (MFX_ERR_MORE_DATA == sts)
        {	
	    if (bEndOfStream)
		break;

	    ReadFrames(&m_mfxBS);
	 }

        else if (MFX_ERR_MORE_SURFACE == sts || MFX_ERR_NONE == sts)
        {
            // find new working surface 
            nIndex = GetFreeSurfaceIndex(m_pmfxSurfaces, m_mfxResponse.NumFrameActual);

            if (INVALID_SURF_IDX == nIndex)

            {
                return MFX_ERR_MEMORY_ALLOC;            
            }
        }
      
        sts = m_pmfxDEC->DecodeFrameAsync(&m_mfxBS, &(m_pmfxSurfaces[nIndex]), 
           &pmfxOutSurface, &syncp);        

        if (MFX_ERR_NONE == sts)
        {
            sts = m_mfxSession.SyncOperation(syncp, DEC_WAIT_INTERVAL);             
        }            

        if (MFX_ERR_NONE == sts)
        {                
            if (m_bExternalAlloc) 
            {
                sts = m_pMFXAllocator->Lock(m_pMFXAllocator->pthis, 
                   pmfxOutSurface->Data.MemId, &(pmfxOutSurface->Data));
                BREAK_ON_ERROR(sts);

                sts = m_FileWriter.WriteNextFrame(pmfxOutSurface);
                BREAK_ON_ERROR(sts);

                sts = m_pMFXAllocator->Unlock(m_pMFXAllocator->pthis, 
                   pmfxOutSurface->Data.MemId, &(pmfxOutSurface->Data));
                BREAK_ON_ERROR(sts);
            }
            else 
            {
                sts = m_FileWriter.WriteNextFrame(pmfxOutSurface);
                BREAK_ON_ERROR(sts);
            }            

            // decoding progress
            _tprintf(_T("Frame number: %d\r"), ++m_nFrameIndex);            
        }

    } //while processing    

    //save the main loop exit status (required for the case of ERR_INCOMPATIBLE_PARAMS)
    mfxStatus mainloop_sts = sts; 

    // means that file has ended, need to go to buffering loop
    IGNORE_MFX_STS(sts, MFX_ERR_MORE_DATA);

    // incompatible video parameters detected, 
    // need to go to the buffering loop prior to reset procedure 
    IGNORE_MFX_STS(sts, MFX_ERR_INCOMPATIBLE_VIDEO_PARAM);

    // exit in case of other errors
    CHECK_RESULT(sts, MFX_ERR_NONE, sts);          
      
    // loop to retrieve the buffered decoded frames
    while (MFX_ERR_NONE <= sts || MFX_ERR_MORE_SURFACE == sts)        
    {        
        if (MFX_WRN_DEVICE_BUSY == sts)
        {
            Sleep(5);
        }

        mfxU16 nIndex = GetFreeSurfaceIndex(m_pmfxSurfaces, m_mfxResponse.NumFrameActual);

        if (INVALID_SURF_IDX == nIndex)
        {
            return MFX_ERR_MEMORY_ALLOC;            
        }

        sts = m_pmfxDEC->DecodeFrameAsync(NULL, &(m_pmfxSurfaces[nIndex]), 
           &pmfxOutSurface, &syncp);

        if (MFX_ERR_NONE == sts)
        {
            sts = m_mfxSession.SyncOperation(syncp, DEC_WAIT_INTERVAL);
        }

        if (MFX_ERR_NONE == sts)
        {
            if (m_bExternalAlloc) 
            {
                sts = m_pMFXAllocator->Lock(m_pMFXAllocator->pthis, 
                   pmfxOutSurface->Data.MemId, &(pmfxOutSurface->Data));
                BREAK_ON_ERROR(sts);

                sts = m_FileWriter.WriteNextFrame(pmfxOutSurface);
                BREAK_ON_ERROR(sts);

                sts = m_pMFXAllocator->Unlock(m_pMFXAllocator->pthis, 
                   pmfxOutSurface->Data.MemId, &(pmfxOutSurface->Data));
                BREAK_ON_ERROR(sts);
            }
            else 
            {
                sts = m_FileWriter.WriteNextFrame(pmfxOutSurface);
                BREAK_ON_ERROR(sts);
            }            

            // decoding progress
            _tprintf(_T("Frame number: %d\r"), ++m_nFrameIndex);          
        }
    } 

    // MFX_ERR_MORE_DATA is the correct status to exit buffering loop with
    IGNORE_MFX_STS(sts, MFX_ERR_MORE_DATA);

    // exit in case of other errors
    CHECK_RESULT(sts, MFX_ERR_NONE, sts);

// if we exited main decoding loop with ERR_INCOMPATIBLE_PARAM we need to send this status 
// to caller
    if (MFX_ERR_INCOMPATIBLE_VIDEO_PARAM == mainloop_sts) 
    {
        sts = mainloop_sts; 
    }

    return sts; // ERR_NONE or ERR_INCOMPATIBLE_VIDEO_PARAM
}

This sample code was tested on a laptop equipped with Intel® Core™ i5 and Intel® HD Graphics chipset. It runs on Windows* 7 with the Intel® Media SDK v1.5, Intel® Integrated Performance Primitives 6.1, and Microsoft* DirectX SDK installed.

Note that the Intel® Media SDK runs only on Microsoft* Windows* Vista* with Service Pack 2, or Microsoft Windows 7 Operating System. Video decoding acceleration requires a system with the Intel® G45 Express Chipset, Intel® GM45 Express Chipset, or Intel® HD Graphics.

To run the sample program, type the program name demux_decode followed by command-line switches -i for the input MP4 stream and -o for the output filename. The optional command-line switches -hw or -d3d indicate using hardware acceleration or Direct3D* 9 surfaces respectively.

  demux_decode -i C:\MediaData\inputstream.mp4 -o ..\demux_decode\outputfile.yuv -hw -d3d

The program displays the results in the following format

Decoding started

Input file      C:\MediaData\mp4\inputstream.mp4
Output file     ..\DataOut\demux_decode\outputfile.yuv

Input video     AVC
Output format   NV12
Resolution      1280x544
Crop X,Y,W,H    0,0,1280,544
Frame rate      23.98
Memory type             d3d
MediaSDK impl           hw
MediaSDK version        1.0

Frame number: 3096
Decoding finished
Press any key to continue . . .

For more details, please refer to the source code of this sample code.

Conclusion

This whitepaper shows how we can use the UMC class (UMC::MP4Splitter) from the Intel® IPP to de-multiplex a MP4/AVC stream, and then send the video data to the Intel Media SDK decode engine. The output from the decode engine can be used to perform pre-processing, or encoding. All these functions are highly optimized for current and future Intel Graphic hardware.

Author

Loc Nguyen is a Software Engineer in the Software and Services Group at Intel Corporation. His interests include network, power, graphics, and media technologies. He received his M.B.A. from the University of Dallas, Texas, and his Master's degree in Electrical Engineering from McGill University, Canada.

References

[1] Intel® Media Software Development Kit, http://www.intel.com/software/mediasdk/

[2] Intel® Integrated Performance Primitives Unified Media Class, Reference Manual, November 2009. Refer to Intel® IPP http://software.intel.com/en-us/intel-ipp/

[3] Intel® Media Software Development Kit (SDK) Integration with Adobe* Premiere Pro*, http://software.intel.com/en-us/articles/intel-media-software-development-kit-sdk-integration-with-adobe-premiere-pro/

[4] ISO/IEC 14496-14:2003. Information technology -- Coding of audio-visual objects -- Part 14: MP4 File Format.

For more complete information about compiler optimizations, see our Optimization Notice.

Comments (0)

Trackbacks (1)

• Twitter Trackbacks for Using UMC De-multiplexer with the Intel® Media Software Development Kit - Intel® Software Network [intel.com] on Topsy.com
  November 17, 2010 4:18 AM PST

Leave a comment  

Name (required)*

Email (required; will not be displayed on this page)*

Your URL (optional)


Comment*