Demuxing with Intel® Media Software Development Kit

Download Article and Source Code

Download Demuxing with Intel® Media Software Development Kit (PDF 626KB)
Download Demuxing Source Code. (ZIP 40KB) (Note: Licensing terms match Media SDK 2.0)

Abstract:This article presents a method that can be used to perform container demuxing (also commonly named splitting) into video streams that can be used by the Intel® Media SDK decoder. The article details how to use demuxing components from audio/video part of Intel® Integrated Performance Primitives (Intel® IPP) samples to create a demuxing solution based on the Intel Media SDK “sample_decode” project.

Introduction

The Intel® Media Software Development Kit (Intel® Media SDK) is a software development library that exposes the media acceleration capabilities of Intel platforms for video decoding, video encoding, and video pre/post-processing. Intel Media SDK helps developers rapidly develop software that accesses hardware acceleration for video codecs with automatic fallback on software if hardware acceleration is not available.

Intel Media SDK is available free of charge and can be downloaded from here: http://www.intel.com/software/mediasdk/ (Gold release targeting current generation of platforms and Beta release targeting features for next generation platforms)

Intel Media SDK is delivered with a wide range of application samples that illustrate how to use the SDK to encode, decode and transcode media to/from elementary video streams. Aside from the binary Microsoft DirectShow* filters supplied with Intel Media SDK, the SDK does not provide the capability of demuxing (also commonly named splitting) media containers into encoded H.264 or MPEG-2 elementary streams understood by the Intel Media SDK.

This article presents a method that can be used to perform demuxing of mp4 (or MPEG-4 part 14) and mpeg containers, detailing how to use demuxing components from the audio/video parts of Intel® Integrated Performance Primitives (Intel® IPP) samples to create a demuxing solution based on the Intel Media SDK “sample_decode” project.

“Demuxing” is the process of, according to container type standards, extracting video frames and audio samples with corresponding time stamps and media metadata that can be used by the decoder component. While audio decoding and audio demuxing is outside the scope of this article, a similar approach could be used to add that functionality.

 

Building the solution described in this article requires the following components:

In addition, the developer also needs Microsoft Visual Studio* and Microsoft Windows* SDK. (Microsoft Visual Studio* 2008 and Microsoft Windows* SDK 7.0 were used when building this solution; we suggest that developers use the same configuration to minimize changes)

*Note: The same solution could easily be migrated to Intel Media SDK 3.0 beta samples.


Overview

The following figure illustrates the process of demuxing container into elementary stream video frames and then decoding the frames. Note that audio decode and demuxing is out of scope of this article. However, a developer could quite easily extend the capability to also handle demuxing into encoded audio samples.


Figure 1 - Overview of Decode - Demuxing process


The solution detailed in this article was created using Intel Media SDK 2.0 with a corresponding video decode console sample, “sample_decode”. The solution also depends on the use of Intel IPP with adjoining audio/video codec samples.

In the sections below we will describe these components in more detail.

 

Intel® Media SDK

Intel® Media SDK supports hardware accelerated and software optimized media libraries for video encode, decode and processing functionality on Intel platforms. The optimized media libraries are built on top of Microsoft DirectX*, DXVA APIs and platform graphics drivers. Intel® Media SDK exposes the hardware acceleration features of Intel® Quick Sync Video built into 2nd generation Intel® Core™ processors. For more information about Intel Quick Sync Video, please see: http://www.intel.com/technology/quicksync/index.htm

The figure below provides a high-level overview of where Intel® Media SDK fits into the software stack.

Intel® Media SDK Applications
Intel® Media SDK API
Intel® Media SDK
Optimized Media Library for CPU
Intel® Media SDK
Optimized Media Library
for
Intel® Processor Graphics
DXVA / DDI Extensions
Graphics Drivers

Figure 2 - Overview of Intel® Media SDK


For extensive details on all the features of Intel Media SDK, please refer to the manual provided with the SDK.

 

Intel® Integrated Performance Primitives (Intel® IPP)

Intel® Integrated Performance Primitives (Intel® IPP) is an extensive library of multi-core-ready, highly optimized software functions for digital media and data-processing applications. Intel IPP offers thousands of optimized functions covering frequently used fundamental algorithms.

The solution described in this article utilizes Intel IPP 7.0 as part of the Intel IPP sample code demuxing components.

The following picture gives an overview of the capabilities of Intel IPP.

Figure 3
Figure 3 - Overview of Intel® IPP


For further details on the Intel IPP licensing terms, please refer to the Intel IPP product page or the following link:
/en-us/articles/intel-integrated-performance-primitives-faq


Intel® IPP Samples

This article uses the audio/video codec samples provided as a separate download with the core Intel® IPP package. Only the “audio-video-codecs” subfolder part of the samples is used in the following solution.

Inside the “audio-video-codecs” subfolder, there is a “doc” folder containing a document with further details on how to use the range of sample components (including the mp4 and mpeg demuxers used in this article).

 

Implementation

In this chapter, we will detail all the steps required to implement MPEG-2 and H.264 demuxing support and how to integrate it with Intel® Media SDK. This includes instructions on how to make the required changes to build projects, library/include file dependencies, and source code, and how to build the Intel® IPP sample components.

*Note that demuxing could also be performed using third-party tools such as ffmpeg, but this naturally allows for much less flexibility. For instance, “ffmpeg -i file.mp4 -vcodec copy -an -vbsf h264_mp4toannexb stream.h264”, will demux a mp4 container into a H.264 elementary video stream.

 

Building Intel® IPP samples demuxer components

First we will build the Intel® IPP audio/video sample components (these components are all in the “UMC” name space) that we will access as libraries in our solution.

Make sure you have installed at least version 7.0 of Intel IPP (or Intel® Composer XE that includes Intel IPP) and the Intel IPP samples.

Also, please set the “IPPROOT” system environment variable to the Intel IPP root folder in your file system. For instance: IPPROOT = C:\Program Files\Intel\ComposerXE-2011\ipp\

Execute the following steps to build the Intel IPP sample components required for the demuxing capability:

  1. Open Command prompt
    (depending on where you execute from, you may have to open in Administrative mode)
  2. Change directory to Intel IPP environment setup folder
    For instance: C:\Program Files\Intel\ComposerXE-2011\ipp\bin\
  3. Execute the Intel IPP environment script
    ippvars.bat <arch>
    (where <arch> is “ia32” or “intel64”)
  4. Change directory to Intel IPP samples audio-video-codecs folder
    <location of Intel IPP samples>\audio-video-codecs\
  5. In this folder you will find a Makefile. Modify the Makefile compile flag M_FLAG and set it to “/MTd” instead of the default “/MD”. (Multi-threaded debug linked library)
    (This is required to match the project settings of the Intel® Media SDK sample_decode project we base the development on)
  6. Run the build script
    For 32 bit: build_ia32.bat cl9
    For 64 bit: build_intel64.bat cl9
    “cl9” indicates the version of Microsoft Visual Studio*. In the example above “9” indicates version 9 of the tool which equals Microsoft Visual Studio* 2008. Modify according to your setup (there is more info on this in \audio-video-codecs\ReleaseNotes.htm).
  7. Note that the build is somewhat time-consuming. After compilation, make sure all the base libraries compiled ok; don’t worry if the application projects failed to compile.
    The binary libraries are created in:
    <location of Intel IPP samples>\audio-video-codecs\_bin\<arch>_cl9\lib\

Set “IPPSAMPLESROOT” system environment variable to the root folder of the Intel IPP samples. For instance: IPPSAMPLESROOT = C:\Program Files\Intel\MediaSDK\ipp_samples Make sure the folder “audio-video-codecs” is located inside the folder of the above path.

 

Integrating demuxer components with decoder

Note that the code below follows the same programming conventions and macros from the existing Intel® Media SDK samples. Please refer to the Intel Media SDK sample code for further details.

For exact details on the code changes, please refer to the provided Microsoft Visual Studio* solution/projects.

 

Changes to Microsoft Visual Studio* “sample_decode” project

Since the following solution is based on the existing “sample_decode” project part of the Intel® Media SDK sample code, please first make a backup copy of the “sample_decode” and “sample_common” project folders before making the changes.

Make sure environment variables IPPROOT and IPPSAMPLESROOT are set as described in the previous chapter.

Please add the following to your Microsoft Visual Studio* project target configuration (or configurations if you plan to compile for several targets) part of the “sample_decode” solution (.sln file).

Add the following to your list of project (sample_common and sample_decode) include directories:

$(IPPSAMPLESROOT)audio-video-codecscodecmpeg4_splinclude
$(IPPSAMPLESROOT)audio-video-codecscodecdemuxerinclude
$(IPPSAMPLESROOT)audio-video-codecscoreumcinclude
$(IPPSAMPLESROOT)audio-video-codecsioumc_ioinclude
$(IPPSAMPLESROOT)audio-video-codecscorevminclude
$(IPPSAMPLESROOT)audio-video-codecscodecspl_commoninclude
$(IPPSAMPLESROOT)audio-video-codecscorevm_plusinclude
$(IPPSAMPLESROOT)audio-video-codecsiomedia_buffersinclude
$(IPPROOT)include

Add the following to your list of project (sample_decode) library directories:
$(IPPSAMPLESROOT)\audio-video-codecs\_bin\ia32_cl9\lib
(Replace ia32_cl9 with appropriate folder name reflecting your architecture and tool version, as described in the previous chapter)

$(IPPROOT)\lib\ia32
(ia32 or intel64 depending on selected architecture)

Add the following to your list of project (sample_decode) library dependencies:

umc.lib
umc_io.lib
mpeg4_spl.lib
demuxer.lib
vm.lib
vm_plus.lib
spl_common.lib
media_buffers.lib
common.lib
ippcore_l.lib
ipps_l.lib

Add the following to your list of project (sample_decode) ignored libraries:

ippcore.lib
ipps.lib

For this specific solution we are using statically linked Intel® IPP libraries which will make the executable a bit larger, but also more portable, compared to a solution using Intel IPP DLLs.

 

Usage scenario / command-line input

To expose the new demuxing feature, we add a new “-split” command line option.

With this new option, we can demux a media container (of type mp4 or mpeg) into video frames that are subsequently decoded by the Intel® Media SDK decoder using the following example command:

sample_decode.exe h264 –i media.mp4 –o out.yuv –split

*Note: The same options available as part of the original decode sample are also available in the new demuxer solution.

The decode target can also be set to “mpeg2” instead of “h264” to process an mpeg type container. For simplicity, MPEG-2 streams are currently only supported in an mpeg container, and H.264 in an mp4 container. The container standards allow for other permutations, but these are outside the scope of this article.

The following code in “sample_decode.cpp” implements support for the new option:

…
else if (0 == _tcscmp(strInput[i], _T("-split")))
{
       pParams->bSplit = true;
}
…

Integration of new SplitterReader class

The demuxing capability is integrated into the exiting sample_decode project by sub-classing the CSmplBitStreamReader class. The new subclass is named SplitterReader , and it implements the complete mp4 and mpeg demuxing functionality using the IPP sample components.

To integrate with the new demuxer class, the following changes are made to “pipeline_decode.cpp”. The code checks whether the demuxing option is enabled and instantiates the appropriate object type:

if(!pParams->bSplit)
	{
		m_pFileReader = new CSmplBitstreamReader();
		sts = m_pFileReader->Init(pParams->strSrcFile);
	}
	else
	{
		m_pFileReader = new SplitterReader();
		sts = static_cast<SplitterReader*>(m_pFileReader)->Init(
					pParams->strSrcFile,
					pParams->videoType);
	}
	CHECK_RESULT(sts, MFX_ERR_NONE, sts);

As can be seen in the code above, the file reader object has changed to a pointer type instead of a directly accessed object in “pipeline_decode.h”.

CSmplBitstreamReader *m_pFileReader;
 

Demuxer class

The majority of the code changes are in the “sample_common” project. The new demuxer class, SplitterReader, is a subclass of the CSmplBitStreamReader and is implemented in the existing “sample_utils.cpp” file.

In the sample_utils.h include file, we must first include all the required header files for demuxer functionality from the Intel® IPP samples audio/video components package.

	#include "umc_structures.h"
	#include "umc_file_reader.h"
	#include "umc_threaded_demuxer.h"
	#include "umc_mp4_spl.h"
	#include "umc_splitter.h"

Next, we need to define the new SplitterReader class as follows:

class SplitterReader : public CSmplBitstreamReader
{
public :
    SplitterReader();
    virtual ~SplitterReader();

    virtual void      Close();
    virtual mfxStatus Init( const TCHAR *strFileName,
				 const mfxU32 nCodecId);
    virtual mfxStatus ReadNextFrame(mfxBitstream *pBS);
protected:
	UMC::Splitter*		m_pSplitter;
	UMC::SplitterParams*	m_pSplitParams;
	UMC::FileReader		m_reader;
	UMC::FileReaderParams	m_frp;
	UMC::SplitterInfo*		m_pSplitterInfo;
	UMC::MediaData*		
	m_pVideoDecSpecInfo;
	UMC::VideoStreamInfo	m_videoInfo;
	UMC::MediaData		m_frameData;
};

From the above code, you can see that we utilize the base, mp4 and mpeg demuxer capabilities from the Intel IPP sample code.

The full demuxer behavior is implemented in the SplitterReader class in “sample_utils.cpp”. There are three key operations involved in using the Intel IPP Samples demuxer components:

  1. Initialize demuxer and fetch container meta data: Generic meta data is retrieved from m_pSplitterInfo (UMC::SplitterInfo) such as rate, duration, container type. Video track-specific data is found in m_videoInfo (UMC::VideoStreamInfo). Decoder-specific bitstream info (if available) is found in m_pVideoDecSpecInfo (UMC::MediaData). Note that this example does not go into detail on how to utilize the above meta data.
  2. Run demuxer: This will cue the demuxer to start parsing the container and filling up the internal demuxer buffers.
  3. Fetch demuxed video frames: This is performed by calling GetNextData on the splitter to retrieve the buffer, then copying it into the Intel Media SDK context bitstream buffer.

The code below shows the actual implementation of these operations used in the context of the SplitterReader class.

SplitterReader::SplitterReader() :
	m_pSplitter(NULL),
	m_pSplitParams(NULL),
	m_pSplitterInfo(NULL),
	m_pVideoDecSpecInfo(NULL)
{
}
SplitterReader::~SplitterReader()
{
        Close();
}

void SplitterReader::Close()
{
        SAFE_DELETE(m_pSplitter);
        SAFE_DELETE(m_pSplitParams);

        m_bInited = false;
}

mfxStatus SplitterReader::Init(	const TCHAR *strFileName,
					            const mfxU32 nCodecId)
{
        UMC::Status umcRes = UMC::UMC_OK;

        Close();

	if(MFX_CODEC_AVC == nCodecId)
	{
		m_pSplitter		= new UMC::MP4Splitter();
		m_pSplitParams	= new UMC::SplitterParams();
	}
	else if(MFX_CODEC_MPEG2 == nCodecId)		//Allocate the appropriate demuxer and demuxer parameter objects
	{
		m_pSplitter		= new UMC::ThreadedDemuxer();
		m_pSplitParams	= new UMC::DemuxerParams();
	}
	else
		return MFX_ERR_UNSUPPORTED;


	// Initialize file reader
	char tempStr[UMC::MAXIMUM_PATH];
	wcstombs(tempStr, strFileName, UMC::MAXIMUM_PATH);
	strcpy((char *)m_frp.m_file_name, tempStr);
	m_frp.m_portion_size = 0;
	umcRes = m_reader.Init(&m_frp);
	CHECK_NOT_EQUAL(umcRes, UMC::UMC_OK, MFX_ERR_UNKNOWN);

	// Initialize splitter
	m_pSplitParams->m_pDataReader		= &m_reader;	// Initialized demuxer
	m_pSplitParams->m_uiSelectedVideoPID	= 0; // Arbitrary ID chosen by user
	m_pSplitParams->m_lFlags			= UMC::VIDEO_SPLITTER;
	m_pSplitParams->m_pMemoryAllocator	= NULL;

	umcRes = m_pSplitter->Init(*m_pSplitParams);
	CHECK_NOT_EQUAL(umcRes, UMC::UMC_OK, MFX_ERR_UNKNOWN);
	
	// Fetch meta data from container (returns pointer to internal structure!)
	umcRes = m_pSplitter->GetInfo(&m_pSplitterInfo);	//fetch container meta data (see above description)
	CHECK_NOT_EQUAL(umcRes, UMC::UMC_OK, MFX_ERR_UNKNOWN);

	// Fetch first video track from splitter
	umcRes = UMC::UMC_ERR_INVALID_STREAM;
	for(Ipp32u i=0; i<(m_pSplitterInfo->m_nOfTracks); i++)
	{
		if(m_pSplitterInfo->m_ppTrackInfo[i]->m_Type & UMC::TRACK_ANY_VIDEO)
		{
			memcpy(&m_videoInfo,
				   m_pSplitterInfo->m_ppTrackInfo[i]->m_pStreamInfo,
				   sizeof(UMC::VideoStreamInfo));
			m_pVideoDecSpecInfo =
                   m_pSplitterInfo->m_ppTrackInfo[i]->m_pDecSpecInfo;
			umcRes = UMC::UMC_OK;
			break;
		}
	}
	CHECK_NOT_EQUAL(umcRes, UMC::UMC_OK, MFX_ERR_UNKNOWN);

	// Kick-start splitter, splitter will start filling up internal buffers
	umcRes = m_pSplitter->Run();							//start demuxer processing
	CHECK_NOT_EQUAL(umcRes, UMC::UMC_OK, MFX_ERR_UNKNOWN);

    m_bInited = true;

    return MFX_ERR_NONE;
}

mfxStatus SplitterReader::ReadNextFrame(mfxBitstream *pBS)
{
	UMC::Status	umcRes		= UMC::UMC_OK;
	size_t		nBytesRead	= 0;
	void*		dataptr;
	size_t		datasize;

	pBS->DataLength = 0;
	pBS->DataOffset = 0;

	// Get frames from splitter to fill up Media SDK bitstream
	while(pBS->DataLength < pBS->MaxLength && umcRes == UMC::UMC_OK)
	{
		umcRes = m_pSplitter->CheckNextData(&m_frameData, 0);	
                //Check if more data available from demuxer
		if(UMC::UMC_OK == umcRes)
		{
			datasize = m_frameData.GetDataSize();
			if(datasize + pBS->DataLength < pBS->MaxLength)
			{
				// fetch next frame from splitter
				umcRes = m_pSplitter->GetNextData(&m_frameData, 0);
				if(UMC::UMC_OK == umcRes) //Get demuxed video frame and copy to Intel Media SDK bit-stream
				{
					dataptr = m_frameData.GetDataPointer();
					datasize = m_frameData.GetDataSize();
					memcpy(pBS->Data + pBS->DataLength,dataptr,datasize);
					pBS->DataLength += static_cast<mfxU32>(datasize);
					nBytesRead += datasize;
				}
			}
			else
			{
				// BS buffer full, exit out of reader
				umcRes = UMC::UMC_ERR_NOT_ENOUGH_BUFFER;
			}
		}
		else if(UMC::UMC_ERR_NOT_ENOUGH_DATA == umcRes) {
			// Splitter is processing data, wait for a bit
			umcRes = UMC::UMC_OK;
			Sleep(5);
		}
	}

    if(0 == nBytesRead)
           return MFX_ERR_MORE_DATA; // Indicates file has been completely processed
    else
           return MFX_ERR_NONE;
}

Note that the above code provides an example on how to use the Intel® IPP sample demuxer components together with the Intel® Media SDK. There are several additional demuxer capabilities a developer can explore, such as repositioning of the content using “SetTimePosition”. For further details, please refer to the Intel IPP samples source code or document located in the “doc” folder under “audio-video-codecs”.

The “audio-video-codecs” (sometimes also named Unified Media Classes, or UMC) provides a wide range of other codec components such as muxers and audio codecs that could be used in connection with Intel Media SDK.

 

Conclusion

In this article, we presented a method to integrate Intel® Media SDK with Intel® IPP samples to enable developers to enhance their Intel Media SDK solutions with demuxing capabilities. A developer can use this method to quickly implement demuxing support into a Intel Media SDK solution.

This is not the only way of integrating demuxing capabilities with Intel Media SDK. There are many other third-party tools or methods that can be used to perform demuxing. However, the method presented provides good flexibility and tight integration.

Download the Demuxing Source Code.(Note: Licensing terms match Media SDK 2.0)

For developer questions on how to use Intel Media SDK in general, please refer to our Intel Media SDK forum on the Intel® Developer Zone site:
/en-us/forums/intel-media-sdk/

References

For more complete information about compiler optimizations, see our Optimization Notice.