Intel® Media Software Development Kit (SDK) Integration with Adobe* Premiere Pro*

by Petter Larsson (Intel Corporation)
Key Contributors: Viacheslav Korobycin, Eric Palmer (Intel Corporation)

Download PDF

Intel® Media Software Development Kit (SDK) Integration with Adobe* Premiere Pro* [PDF 1MB]


Introduction

Intel® Media Software Development Kit (SDK) is a software development library that exposes the media acceleration capabilities of Intel platforms for decoding, encoding and video prepocessing. The SDK helps developers rapidly develop software accessing hardware acceleration for video codecs with automatic fallback on software if hardware acceleration is not available. Intel® Media SDK also supports future integrated graphics products and discrete graphics hardware products that will feature hardware accelerated encode and enhanced hardware accelerated decode.

Intel® Media SDK is packaged with many simple application samples that illustrate how to use the SDK to encode, decode and pre-process media.

The Intel® Media SDK is available for download here: www.intel.com/software/mediasdk

The following article is a case study on how the Intel® Media SDK was integrated with a leading video authoring application, Adobe* Premiere Pro* CS4 for Microsoft Windows*. The paper provides details on the specific integration points, key design considerations and supplemental component usage.

Note that this is by no means a complete integration exposing all features of the Intel® Media SDK or the Adobe* Premiere Pro* SDK, instead the implementation should be considered a prototype that could be used as a base for further development.

For exact implementation details please refer to the supplied source code.

Overview


Intel® Media SDK

Intel® Media SDK is a software development library that exposes the media acceleration capabilities of Intel platforms for decoding, encoding and video prepocessing. The SDK exposes a uniform API shielding the developer from platform specific code and allows easy access to hardware acceleration if available.
The SDK currently supports an optimized media library for software based encode and decode functionality, and an optimized media library for hardware decode using Intel® Integrated Graphics (IIG) chipsets such as the Intel® G45 Express chipset. Future Intel® products featuring Discrete Graphics will also be supported. The optimized media libraries are built on top of Microsoft DirectX* Video Acceleration (DXVA) API and platform graphics drivers.

Intel® Media SDK Sample Applications

Intel® Media SDK API


Intel® Media SDK – Optimized Media Library for CPU

Intel® Media SDK – Optimized Media Library for Intel® Integrated & Discrete Graphics

DXVA / DDI Extensions

Graphics Drivers

Figure 1: Intel® Media SDK Overview

The Intel® Media SDK has a range of features

  • Codec support
    • Encode: MPEG-2, H.264 (both in SW only)
    • Decode: MPEG-2, H.264 and VC1 (all in SW and HW)
  • Pre Processing support (VPP): Deinterlacing, Denoising, Resizing, Color conversion, Inverse telecine, Scene detection
  • Codec profiles
    • MPEG-2: Simple, Main, High
    • H.264/AVC: Baseline (encoding), Main, High
    • VC1: Simple, Main, Advanced
  • Progressive and Interlace frame types
  • CBR and VBR rate control

Some of these features will be utilized in the prototype described in the following chapters. The prototype code is based on encoder and decoder sample code provided by the SDK, utilizing many of the common components and utility functions.



Adobe* Premiere Pro* CS4 SDK

Adobe* Premiere Pro* CS4 SDK offers developers the ability to enhance and extend the functionality of the Adobe*Premiere Pro* product.

The SDK exposes common interfaces that are used by Adobe* Premiere Pro*, After Effects*, SoundBooth*, Encore*, and Premiere* Elements. The SDK includes sample code and documentation for importers, exporters, recorders, compilers, players, transitions, video filters, and device controllers allowing the developer to create a wide range of plug-ins.

The prototype detailed in the following chapters is based on the Importer and Exporter plug-in sample code provided as part of the SDK samples.

The SDK can be downloaded here: http://www.adobe.com/devnet/premiere/sdk/cs4/


Intel® Integrated Performance Primitives (Intel® IPP)

Intel® Integrated Performance Primitives (Intel® IPP) is an extensive library of multicore-ready, highly optimized software functions for digital media and data-processing applications. Intel® IPP offers thousands of optimized functions covering frequently-used fundamental algorithms.

Two components from the Intel® IPP package were utilized by the prototype

  • Image Color Conversion functions from Image & Video processing library
  • Muxer and Demuxer functionality from the Intel® IPP sample package (referred to as Unified Media Classes (UMC) but also denoted Audio Video Codec samples or Media Processing samples)

Details of Intel® IPP can be found here: /en-us/intel-ipp/

Intel® IPP revision 6.1.1 with samples were used as part of the prototype.


Prototype Overview

The below figure illustrates the components utilized as part of the integration

Adobe* Premiere Pro* CS4

Adobe* Premiere Pro* CS4 SDK

Exporter plug-in (H.264/MPEG-2 encoding)

Importer plug-in (H.264 decoding)

Importer plug-in (MPEG-2 decoding)

Intel® Media SDK Integration Layer
- UMC: Muxer/Demuxer
- Intel® IPP: Color conversion
Misc: Memory management, File access etc.

Intel® Media SDK API

Figure 2: Overview: Intel® Media SDK / Adobe* Premiere Pro* CS4 integration

The prototype is implemented as three Adobe* Premiere Pro* plug-ins, a combined H.264/MPEG-2 encoder plug-in (exporter), a H.264 decoder plug-in (importer), and a MPEG-2 decoder plug-in (importer).

The exporter plug-in is accessible from the File menu in Adobe* Premiere Pro* via the item “Export->Media”. The implemented encoder options are configurable via the Export Setting dialog which controls the workload sent for encoding by Adobe* Media Encoder.

The importer plug-ins are accessible from the File menu in Adobe* Premiere Pro* via the item “Import” which opens up a file dialog for selection of encoded media containers.
Integrating the Adobe* Premiere Pro* SDK and the Intel® Media SDK requires use of some additional functionlity provided by Intel® IPP to perform operations such as image color conversion and muxing demuxing video stream to/from the supported media containers, mp4 and mpeg.

The prototype was built using Microsoft* Visual Studio* 2005 and aside from the components described above the following Microsoft packages were also used

  • Microsoft Windows* SDK v6.0
  • Microsoft DirectX* SDK

Prototype Implementation

This chapter describes key integration points and design consideration for the integration of Intel® Media SDK with Adobe* Premiere Pro*.

For specific source code details please refer to the source code: download zip file (119K)

The prototype was created using release 1.0 of the Intel® Media SDK and CS4.1 release of the Adobe* Premiere Pro* SDK. Code changes might be required to adapt to recent changes to these SDKs and recent releases of Intel® IPP libraries. Also note that the encoder and decoder implementation are by no means complete. For instance, error condition handling is not complete and not all features of the Adobe* Premiere Pro* SDK Import and Export types are used. To support extended common functionality exposed by the Adobe* Premiere Pro* SDK or Intel® Media SDK additional development is required.

Since we did not want to overlap existing file extension Importer triggers in Adobe* Premiere Pro* the actual encoded media file extensions used are “sda” and “sdb” for H.264 (mp4 container) and MPEG-2 (mpeg container) respectively. The files can be renamed “*.mp4” and “*.mpeg” respectively if the user needs to import encoded media using other default Adobe* Premiere Pro* importer or other media application.



Code Architecture Overview

The prototype was developed as three separate projects. One project for the encoder plug-in based on the Adobe* Premiere Pro* SDK Exporter plug-in sample code. Two projects for each of the decoder plug-ins based on the Adobe* Premiere Pro* SDK Importer plug-in sample code. The encoder and decoder plug-in utilizes common Intel® Media SDK source code for memory allocation and common libraries from Intel® IPP, Microsoft DirectX* and Intel® Media SDK.

Common components used by Exporter and Importers

Source code
(from Intel® Media SDK samples)

  • base_allocator (h,c)
  • d3d_allocator (h,c)
  • sysmem_allocator (h,c)

Dependencies/
Includes

  • Intel® Media SDK
  • Microsoft Windows* SDK v6.0
  • Microsoft DirectX* SDK
  • Intel® IPP 6.1.1 with samples including UMC
  • Adobe* Premiere Pro* CS4 SDK

Libraries/
DLLs

  • Intel® IPP
    • ippcorel.lib
    • ippsemerged.lib
    • ippsmerged.lib
    • ippdcemerged.lib
    • ippdcmerged.lib
    • ippccemerged.lib
    • ippccmerged.lib
  • Intel® IPP Samples - UMC
    • umc_io.lib
    • vm.lib
    • umc.lib
    • media_buffers.lib
    • common.lib
  • Microsoft DirectX*
    • dxva2.lib
    • d3d9.lib
  • Intel® Media SDK
    • libmfx.lib
    • libmfxsw32.dll
    • libmfxhw32.dll

Exporter

Importers

Source code

  • Integration code
    • IntelMSDKInteg (h,c)
  • Adobe* Premiere Pro* CS4 SDK interface implementation
    • IntelMSDKExport (h,c)
    • IntelMSDKExport_Params (h,c)

Source code

  • Integration code
    • IntelMSDKInteg (h,c)
    • Frame_constructors (h,c) (for h.264)
  • Adobe* Premiere Pro* CS4 SDK interface implementation
    • SDK_File_Import (h,c)
    • SDK_Async_Import (h,c)

Libraries/DLLs

  • Intel® IPP Samples - UMC
    • mpeg4_mux.lib
    • mpeg2_mux.lib

Libraries/DLLs

  • Intel® IPP Samples - UMC
    • mpeg4_spl.lib (for h.264)
    • demuxer.lib (for mpeg-2)
    • spl_common.lib

Table 1: Integration code architecture overview

As can be seen from the table above, the exporter and importer plug-in source are divided into two parts for simplicity. First part, Adobe* Premiere Pro* SDK interface implementation, based on the exporter/importer samples code from the SDK. The second part provides the integration code gluing together the SDKs using Intel® Media SDK utility functions and features from Intel® IPP (UMC samples).


Building UMC sample libraries from IPP samples
Use of the required prototype components are pretty straightforward except for the use of the UMC components part of Intel® IPP samples which requires initial build setup. To build the required UMC libraries used by the importer and exporter plug-ins, use the following procedure:
1. In the downloaded Intel® IPP samples package navigate to ".../audio-video-codecs/"
2. In the Makefile make sure to set the M_FLAG to /MTd (to ensure compile with multithreaded runtime debug library)
3. Build the entire UMC package by executing "build32.bat d8"
4. This will result in a set of libraries in ".../audio-video-codecs/_bin" that will are linked in by the prototypes



Encoder implementation
The encoder is implemented as an Adobe* Premiere Pro* Exporter plug-in type supporting both encoding to H.264 (mp4 container) and MPEG-2 (mpeg container).

The exporter plug-in (.prm file) is statically linked with no dependencies on DLLs except for the Intel® Media SDK device implementation DLLs (libmfxsw32.dll or libmfxhw32.dll). Users need make sure these DLLs are in the system path. Additionally, users need to make sure the plug-in is copied to both the Adobe* Premiere Pro* and Adobe* Media Encoder plug-in folders to ensure automatic loading upon application start.

The below screen shot shows the Intel® Media SDK encoder configuration options exposed via the Adobe* Premiere Pro* Export dialog.


Figure 3: Adobe* Premiere Pro* CS4 Export UI for Intel® Media SDK

As the screen shot illustrates, the following encoding parameters are exposed.

Codec Type (H.264 or MPEG-2)

Frame size

Pixel Aspect Ratio

Video quality

Bit rate (average and max (for VBR))

Rate control (VBR or CBR)

# of encoding threads

Encoding Device (HW or SW)

Memory Allocation Pattern (RAM or D3D)

Table 2: Supported encoding parameters

When the user commits the selected encoding parameters the task of the actual encoding is transferred to Adobe* Media Encoder which opens up a dialog similar to the screen shot below.


Figure 4: Adobe* Media Encoder Dialog

To encode the media using the prototype the user selects “Start Queue” which effectively kicks off the encoding process, for which the implementation is described in the following chapter.


Adobe* Premiere Pro* SDK encoder integration
The Adobe* Premiere Pro* SDK events are listed in the order in which they are invoked. Events not listed below have been left unchanged from the sample Export plug-in code. The events are dispatched from the plug-in entry point, “DllExport PREMPLUGENTRY xSDKExport”

  • exSelStartup: Handles initial Exporter plug-in configuration
    • Initialize use of static Intel® IPP libraries

      ippStaticInit()

    • Set Exporter identifier name

      fileTypeName = "Intel Media SDK Encode"

    • Disable Audio export since Intel® Media SDK does not support Audio encode or decode

      canExportAudio = False

  • exSelGenerateDefaultParams: Define configurable parameter groups and parameters that will be accessible thru the Export user interface. The process includes initializing the parameters with default values
    • A range of encoding parameters are supported. Refer to "Table 2: Supported encoding parameters"
    • Default frame rate and resolution is deduced from media selected for encode
    • Configurable parameters are exposed in groups via the of the "ExportParamSuite" For instance:

      AddParamGroup(exporterPluginID, mgroupIndex, ADBEVideoTabGroup,
      ADBEVideoCodecGroup, VIDEO_CODEC_PARAM_GROUP_NAME,
      kPrFalse, kPrFalse, kPrFalse);

      AddParam(exporterPluginID, mgroupIndex, ADBEVideoCodecGroup,
      &codecParam);


  • exSelPostProcessParams:Define parameter name identifiers and allowed values for each of the parameters defined from in "exSelGenerateDefaultParams".
    • Using "ExportParamSuite" set parameter name identifiers. For instance:

      SetParamName(postProcessParamsRecP->exporterPluginID, 0,
      ADBEVideoCodecGroup, VIDEO_CODEC_PARAM_GROUP_NAME);

    • Set allowed values or range using the "ExportParamSuite". For instance

      ClearConstrainedValues(postProcessParamsRecP->exporterPluginID, 0,
      ADBEVideoCodec);

      AddConstrainedValuePair(postProcessParamsRecP->exporterPluginID, 0,
      ADBEVideoCodec, &tempCodec, codecStrings[i]);


  • exSelQueryOutputSettings: Assembles encoding parameters used for preview by Adobe* Media Encoder. Also calculate combined audio + video bitrate
    • Parameter settings are retrieved via "ExportParamSuite". For instance:

      GetParamValue(exporterPluginID, mgroupIndex, ADBEVideoWidth, &width);

  • exSelGetParamSummary: Assemble basic parameter settings displayed in the "Export Settings" box. Parameters exposed: Resolution, frame rate, pixel aspect ratio, video quality, bit rate and rate control
    • Parameter settings are retrieved via "ExportParamSuite". For instance:

      GetParamValue(exporterPluginID, mgroupIndex, ADBEVideoWidth, &width);

  • exSelValidateParamChanged: Validate parameters changed by the user via the UI. This allows updating of parameters that may have interdependencies, for instance VBR/CBR rate control and bit rate and max bit rate.
    • Parameter settings are retrieved and modified via "ExportParamSuite". For instance:

      GetParamValue(validateParamChangedRecP->exporterPluginID,
      validateParamChangedRecP->multiGroupIndex,
      ADBEVideoRateControl, &rateControl);

      ChangeParam(validateParamChangedRecP->exporterPluginID,
      validateParamChangedRecP->multiGroupIndex,
      ADBEVideoBitRateMax, &rateValues);

  • exSelQueryExportFileExtension: Set media container file extension depending on codec type chosen. "mp4" type/container is used for H.264 and "mpeg" type/container is used for MPEG-2. Since we did not want to overlap existing file extension Importer triggers in Adobe* Premiere Pro* the actual file extensions used are "sda" and "sdb" for H.264 and MPEG-2 respectively.
  • exSelExport: This event is triggered by the user selecting to start encoding using Adobe* Media Encoder CS4 which is started when the user commits the export settings from the Adobe* Premiere Pro* Export UI.
    • The details of this process in handled separately in the following chapter

The code is implemented in “IntelMSDKExport(c.h)” and “IntelMSDKExport_Params(c.h)”.



Encoding process
The encoding process in triggered by the Adobe* Exporter plug-in event “exSelExport”.
The different tasks are listed in order and prefixed with the actual component involved: Intel® IPP, UMC, Adobe* SDK and Intel® Media SDK.

  • UMC: Depending on used codec create MP4 of MPEG2 muxer
  • pMuxer = new

    • Adobe* SDK: Fetch location to where encoded file should be written using "ExportFileSuite"

      GetPlatformPath(compileInfoP->fileObject, &filenamelength, filename);
  • UMC: Initialize the muxer "FileWriter" component which handles writing of each frame to the media container. Using the integration utility class "CWriter"
  • Intel® Media SDK: Initialize Intel® Media SDK VPP parameters using the "InitMfxVppParams" integration utility function

    • Fetch configured plug-in parameters via the "ExportParamSuite"
    • Reset VPP algorithm settings
    • Configure input and output Intel® Media SDK VPP parameters
      • Memory pattern selected
      • Color format (Input = YV12, Output = NV12)
      • Convert and set to Intel® Media SDK formatted frame rate
      • Frame Resolution selected
  • Intel® Media SDK: Initialize Intel® Media SDK encoder parameters using the "InitMfxEncoderParams" integration utility function
    • Fetch configured plug-in parameters via the "ExportParamSuite"
    • Configure Intel® Media SDK encoder parameters
      • Code type (H.264/AVC or MPEG-2) and profile
      • Memory pattern selected
      • Video quality selected
      • Bit rate selected
      • Rate control selected
      • Convert and set to Intel® Media SDK formatted frame rate
      • Number if selected encoding threads
      • Frame Resolution selected
      • Color format (NV12)
  • Intel® Media SDK: Initialize Intel® Media SDK resources using the "InitResources" integration utility function
    • Create frame processor: Initialize the Intel® Media SDK session selecting the encoding device to be used (currently only SW is supported). Create Intel® Media SDK encoder and VPP frame processors. Prepare encoding and VPP surfaces

      mfxSession.Init((mfxIMPL)encodingDevice.value.intValue, &version);

      pmfxENC = new MFXVideoENCODE(pProcessor->mfxSession);

      pmfxVPP = new MFXVideoVPP(pProcessor->mfxSession);
    • Initialize memory allocator: Determine the number of surfaces required for encode and VPP. Using the Intel® Media SDK sample utility functions to setup system or D3D memory allocation

      pmfxENC->QueryIOSurf(pEncoderParams, &EncRequest);

      pmfxVPP->QueryIOSurf(pVppParams, VppRequest);

      pMfxAllocator = new D3DFrameAllocator;
      or
      pMfxAllocator = new SysMemFrameAllocator;

      pMfxAllocator->Init(pAllocator->pAllocatorParams);

      pMfxAllocator->Alloc(pAllocator->pMfxAllocator->pthis,
      &EncRequest, &pAllocator->EncResponse);
    • Initialize frame processor: Initialize encoder and VPP frame processors

      pmfxENC->Init(pEncoderParams);

      pmfxVPP->Init(pVppParams);
    • Initialize task pool: Setup task pool for concurrent frame encoding using the Intel® Media SDK sample utility TaskPool class

      pTaskPool->Init(pEncoderParams, nTasks,
      pResources->pProcessor,
      pResources->pDstFileWriter);

    • Adobe* SDK: Create video renderer component using the "SequenceRenderSuite", specifying number ticks per frame

      MakeVideoRenderer(compileInfoP->exporterPluginID, &mySettings->videoRenderID,
      ticksPerFrame.value.timeValue);

  • Adobe* SDK: Set the specified render parameters, "SequenceRender_ParamsRec", used during request of uncompressed video frame via "RenderVideoFrame" operation inside main encoding loop
    • UMC: Set UMC muxer parameters (with one single video track) and initialize the previously created muxer

    pMuxer->Init(&muxerParams);

    • Enter main encoding loop handling Intel® Media SDK processing of each video frame retrieved via Adobe* SDK API. Encoding is performed in asynchronous mode.
      • Intel® Media SDK: Get free frame surface for encoder and VPP (if required)

        GetFreeSurface(Allocator.pEncoderSurfaces,
        Allocator.EncResponse.NumFrameActual,
        &pEncoderSurface);
      • Fetch, render and process frame using RenderAndFetchVideoFrame()
        • Intel® Media SDK: Lock memory frame (applicable when using D3D memory surface)
        • Adobe* SDK: Render single uncompressed video frame to buffer and fetch additional frame info using SequenceRenderSuite and PPixSuite

          RenderVideoFrame(...),GetPixels(...), GetRowBytes(...), GetPixelFormat(...)
        • Intel® IPP: Perform color conversion from BGR format received from Adobe* SDK to YUV420 format required by Intel® Media SDK

          ippiBGRToYUV420_8u_AC4P3R(...)
        • Adobe* SDK: Dispose of frame buffer

          Dispose(...)
        • Intel® Media SDK: Unlock memory frame (applicable when using D3D memory surface)

      • Intel® Media SDK: Perform VPP if required

        RunFrameVPPAsync(pVppSurface, pEncoderSurface, NULL, &syncPointVpp);

      • Intel® Media SDK: Get free transform frame from task pool. The task pool is handling the synchronization between VPP and encode.

        TaskPool.GetFreeTransformFrame(&TransformFrame);

      • Intel® Media SDK: Encode frame

    EncodeFrameAsync(NULL, pEncoderSurface, TransformFrame.pmfxBS,
    TransformFrame.psyncP);

    • Adobe* SDK:Update Adobe* Media Encoder UI progress bar using ExportProgressSuit

      UpdateProgressPercent(compileInfoP->exporterPluginID, progress);

    • Intel® Media SDK: After fetching and feeding all frames to Intel® Media SDK, process remaining buffered frames in task pool
    • UMC: Finalize, close and free muxer stream

      pMuxer->PutEndOfStream(iVideoPin);
      pMuxer->Close();
    • Intel® Media SDK: Release Intel® Media SDK resources such as task pool, memory allocator etc.

      WipeResources(&Resources);
    • Adobe* SDK: Release video renderer

      ReleaseVideoRenderer(compileInfoP->exporterPluginID,
      mySettings->videoRenderID);


    The code is implemented in “IntelMSDKExport(c.h)” and “IntelMSDKInteg(c.h)”.



    Decoder implementation

    The decoder is implemented as two separate Adobe* Premiere Pro* Importer plug-ins supporting decoding of H.264 (mp4 container) and MPEG-2 (mpeg container).

    The importer plug-in (.prm file) is statically linked with no dependencies on DLLs except for the Intel® Media SDK device implementation DLLs (libmfxsw32.dll or libmfxhw32.dll). Users need make sure these DLLs are in the system path. Additionally, users need to make sure the plug-ins are copied to both the Adobe* Premiere Pro* and Adobe* Media Encoder plug-in folders to ensure automatic loading upon application start.

    Since Adobe* Premiere Pro* Importers integrate seamlessly with the application there are no new dialogs exposed to the user. After importing a file, it shows up in the Project panel with corresponding thumbnail representing the first frame of the imported media. The can then play back or operate on the imported media using normal Adobe* Premiere Pro* workflow.

    Adobe* Premier Pro* SDK decoder integration
    The Adobe* Premiere Pro* SDK events are listed in the order in which they are invoked. Events not listed below have been left unchanged from the sample Export plug-in code. The events are dispatched from the plug-in entry point, “DllExport PREMPLUGENTRY xImportEntry”

    The different tasks are listed in order and prefixed with the actual component involved: Intel® IPP, UMC, Adobe* SDK and Intel® Media SDK.

  • imInit: Initialize Importer plug-in
    • Setup basic plug-in properties
    • Initialize use of static Intel® IPP libraries

      ippStaticInit()
  • imGetIndFormat: Set importer plug-in format name and file extension. To avoid conflict with other MPEG-2 and H.264 decoders two new file name extensions were defined. H.264 container media files use the "sda" extension and MPEG-2 container media files use the "sdb" extension
  • imOpenFile8: Setup demuxing functionality and fetch media container meta data
    • Intel® Media SDK: Create basic Intel® Media SDK resources
    • UMC: Depending on media type, create MP4 or MPEG-2 demuxer (splitter). For instance:

      pSplitter = new UMC::MP4Splitter();
    • UMC: Create and initialize splitter file reader

      pReader = new UMC::FileReader();
      pReader->Init(&readerParams);
    • UMC: Initialize splitter

      pSplitter->Init((*localRecH)->pResources->splitParams);
    • UMC: Fetch container meta data and track info and run splitter

      pSplitter->GetInfo(&((*localRecH)->pResources->pSplitterInfo));
      pSplitter->Run();
  • imGetInfo8: Setup basic Adobe* SDK decoding parameters and initialize Intel® Media SDK
    • Intel® Media SDK: Initialize Intel® Media SDK using integration utility function "InitMSDK"

      • Initialize the Intel® Media SDK session selecting the encoding device to be used (automatic selection of HW if available, fallback on SW)

        mfxSession.Init(MFX_IMPL_AUTO, &version);

      • Create Intel® Media SDK decoder

        pmfxDEC = new MFXVideoDECODE(pResources->pDecoder->mfxSession);

      • Specify the decoder type: H.264 (MFX_CODEC_AVC) or MPEG-2 (MFX_CODEC_MPEG2)
      • Initialize bit stream using integration utility function "InitMfxBitstream", holding encoded the video stream data read via the splitter
      • Fetch video stream header data using splitter. Integrating Intel® Media SDK with the MP4 splitter currently requires reformatting the data using the supplied "CAVCFrameConstructor" class
      • Initialize Intel® Media SDK decoder parameters using "InitMfxParams" integration utility function. Decode video stream header and specify memory type to be used

        pmfxDEC->DecodeHeader(pResources->pBitstream,
        &pResources->mfxParamsVideo);
      • Initialize Intel® Media SDK memory (system or D3D memory) allocator using "CreateAllocator" integration utility function
      • Allocate frames using "AllocFrames" integration utility function. Allocated and setup decode surfaces
      • Initialize Intel® Media SDK decoder

        pmfxDEC->Init(&pResources->mfxParamsVideo);
    • Adobe* SDK: Configure plug-in with video parameters fetched from media stream. Calculate video duration and number of total frames
  • imGetIndPixelFormat:

      • Adobe* SDK: Request preferred pixel format that Adobe* Premiere Pro* should process. To enable straightforward color conversion path "PrPixelFormat_UYVY_422_8u_601" and "PrPixelFormat_BGRA_4444_8u" is used.
  • imCreateAsyncImporter:
      • This event creates an asynchronous handler for decoding
      • Asynchronous events are picked up by the "PREMPLUGENTRY xAsyncImportEntry" function dispatcher, handling frame decoding operations. The frame decoding process is handled in the following section
  • imQuietFile/imCloseFile:
      • Intel® Media SDK: Free Intel® Media SDK decoder resources
      • UMC: Dispose of the splitter

    The code is implemented in “SDK_File_Import(c.h)”, “SDK_Async_Import(c.h)” and “IntelMSDKInteg(c.h)”


    Decoding process
    The decoding process in triggered by the Adobe* SDK asynchronous importer event “aiGetFrame”.
    The different tasks are listed in order and prefixed with the actual component involved: Intel® IPP, UMC, Adobe* SDK and Intel® Media SDK. Events not listed have been left unchanged or not implemented from the sample Import plug-in code.

  • aiGetFrame:
      • Adobe* SDK: Determine the requested frame number and set delivered frame size to same as Intel® Media SDK decoded frame
      • Determine if frame is ready to be delivered to Adobe* Premiere Pro* (3 possible scenarios)
        • 1. Adobe* SDK: If frame is available in cache return the frame pointer

          PPixCacheSuite->GetFrameFromCache ((*localRecH)->importerID, 0,
          (long)theFrame, 1,
          inFrameRec->inFrameFormats,
          &pFrameReq->thePPH);
        • 2. Decoded frame is available
          • Adobe* SDK: Get target buffer for decoded frame storage

            PPixSuite->GetPixels(pFrameReq->thePPH,
            PrPPixBufferAccess_WriteOnly,
            &pFrameBuffer);
          • Intel® IPP: Convert uncompressed format from Intel® Media SDK (NV12) into format requested by Adobe* Premiere Pro* using Intel® IPP color conversion function, such as "ippiYCbCr420ToCbYCr422_8u_P2C2R"
          • Adobe* SDK: Add frame to Adobe* Premiere* cache. Frame is now ready for rendering by Adobe* Premiere Pro*

            PPixCacheSuite->AddFrameToCache ((*localRecH)->importerID,
            0, pFrameReq->thePPH,
            pFrameReq->frameNum);


        • 3. Adobe* SDK: No decoded frames available. Adobe* SDK will ask for reading additional frames via "aiInitiateAsyncRead" event

    • aiInitiateAsyncRead:
        • Adobe* SDK: Determine the requested frame number and set delivered frame size to same as Intel® Media SDK decoded frame
        • Intel® Media SDK/UMC: Check if requested frame is the next frame in decoded stream. If not perform repositioning using the utility function. For instance, repositioning to earlier frame requires UMC Splitter operations Close(), Init() and Run(), as well as Intel® Media SDK decoder Reset()
        • Adobe* SDK: Create pixel map buffer container

          PPixCreatorSuite->CreatePPix(&tempPPH, PrPPixBufferAccess_WriteOnly,
          pFrameFormat->inPixelFormat, &theRect);
        • Enter main decoding loop handling frame reading using splitter and frame decode using Intel® Media SDK

          • Intel® Media SDK/UMC: Fetch frame data from splitter filling up the Intel® Media SDK bitstream buffer (for the prototype the buffer was arbitrarily set to 300KB). Integrating Intel® Media SDK with the MP4 splitter currently requires reformatting the data using the supplied "CAVCFrameConstructor" class

            pSplitter->GetNextData(&videoData, 0);
          • Intel® Media SDK: Get free surface using integration utility function "GetFreeSurfaceIndex"
          • Intel® Media SDK: If enough data is available in Intel® Media SDK bitstream, asynchronously decode next frame

            pDecoder->pmfxDEC->DecodeFrameAsync(
            (*localRecH)->pResources->pBitstream,
            &((*localRecH)->pResources->pAllocator->pSurfaces[nIndex]),
            &pFrameReq->pFrameSurface,
            &((*localRecH)->pResources->syncp));
          • Intel® Media SDK: If frame surface is ready (frame is decoded), exit the decoding loop, otherwise continue decode. Decoded frames are synchronized and handled by "aiGetFrame"

    The code is implemented in “SDK_Async_Import(c.h)”.

    Conclusion

    Throughout this paper we have presented the steps involved in integrating Intel® Media SDK with Adobe* Premiere Pro*. For specific implementation details please refer to the source code: download zip file (119K).

    Using the above SDKs (reusing sample code) and components we found that it was quite straightforward to implement Intel® Media SDK encoder and a decoder plug-ins that exposes the flexibility and performance of Intel® Media SDK to mainstream video authoring application such as Adobe* Premiere Pro*.

    To speed up development and performance we used a couple of readily available components from the Intel® IPP (including UMC framework samples). Alternatively, a developer could choose to use other ways of handling muxing/demuxing and color conversion. For instance, the Intel® Media SDK package also provides muxing/demuxing samples using Microsoft* DirectShow.

    Aside from speeding up development, Intel® Media SDK also future proofs the implementation resting on the SDKs support for upcoming platforms with features such as hardware encode and improved performance.

    Interesting future extension on the topic of this paper might include support for audio encode/decode with muxing/demuxing, support for frame indexing to improve repositioning performance, transcoding feature etc.

    References

    如需更全面地了解编译器优化,请参阅优化注意事项