Intel® GPUs contain fixed function hardware to accelerate video encode, decode, and frame processing, which can now be used with a variety of interfaces. Media SDK and Media Server Studio provide great performance with an API designed around delivering full hardware capabilities that is portable between OSes. However, there is a big limitation: the Media SDK API only processes video elementary streams. FFmpeg is one of the most popular media frameworks. It is open source and easily expandable. Because of this it has a very wide range of functionality beyond just codecs: muxing and demuxing(splitting), audio, network streaming, and more. It is straightforward to extend FFmpeg with wrappers for Intel® HW acceleration. Various forms of these wrappers have existed for many years, and they provide important ease of use benefits compared to writing encode/decode code directly with the Media SDK API. However, the tradeoff for this ease of use is that performance is still left on the table. To get the best of both worlds – full performance and access to the full range of capabilities in FFmpeg – a hybrid approach is recommended.
Intel® provides several ways for you to use hardware acceleration in FFmpeg.
This article provides important updates to the 2012 article. It describes the process to use the FFmpeg libraries on Ubuntu 16.04. The example code will be based on our tutorial code so the user will have a better view on how the FFmpeg API is integrated with the media pipeline. The example code will also update the deprecated FFmpeg API so it is synced with the latest FFmpeg releases.
The project to run the tutorial has the following file structure:
|simple_decode_ffmpeg.cpp is the Media SDK application to create a simple decode pipeline and call the function defined in ffmpeg_utils.h to hook up the demux APIs of FFmpeg library|
|simple_encode_ffmpeg.cpp is the Media SDK application to create a simple encode pipeline and call the ffmpeg adaptive function defined in ffmpeg_utils.h to hook up with the mux APIs of FFmpeg library.|
|The API in these files defines and implements the API to initialize, execute and close the mux and demux functions of the FFmpeg library.|
This is the built FFmpeg libraries, the libraries involved are libavformat.so, libavcodec.so and libavutil.so
PATH="$HOME/bin:$PATH" PKG_CONFIG_PATH="$HOME/ffmpeg_build/lib/pkgconfig" ./configure \ --prefix="$HOME/ffmpeg_build" \ --pkg-config-flags="--static" \ --extra-cflags="-I$HOME/ffmpeg_build/include" \ --extra-ldflags="-L$HOME/ffmpeg_build/lib" \ --extra-libs=-lpthread \ --bindir="$HOME/bin" \ --enable-gpl \ --enable-libass \ --enable-libfdk-aac \ --enable-libfreetype \ --enable-libmp3lame \ --enable-libopus \ --enable-libtheora \ --enable-libvorbis \ --enable-libvpx \ --enable-libx264 \ --enable-libx265 \ --enable-nonfree \ --enable-shared \ --enable-pic \ --extra-cflags=-fPIC
# _build/simple_decode_ffmpeg ~/Downloads/BigBuckBunny_320x180.mp4 out.yuv
The command generates 2 output files: out.yuv--the raw video stream; audio.dat--the raw audio PCM 32bit stream.
# _build/simple_encode_ffmpeg -g 320x180 -b 20000 -f 24/1 out.yuv out.mp4
The command reads the raw audio with the name "audio.dat" by default.
uname -r 4.4.0
In the installation instruction, the kernel 4.4 was patched, this implies the driver update to access the media fixed functions. If the command doesn't show the expected kernel version, user has to switch the kernel at the boot time at the grub option menu, to show the grub menu, refer to this page.
|container with codecs||sample_decode_ffmpeg||sample_encode_ffmpeg|
|.mp4||(h.264/hevc/MPEG2, aac)||(h.264, aac)|
|.mkv||(h.264/hevc/MPEG2, ac3)||(h.264, ac3)|
|.ts||(h264/hevc, ac3)||(MPEG2, aac)|
|.mpg, mpeg||(MPEG2, ac3)||(MPEG2, aac)|
./sample_multi_transcode -i::h264 test_stream.264 -o::h264 out.264 Multi Transcoding Sample Version 18.104.22.1688 libva info: VA-API version 0.99.0 libva info: va_getDriverName() returns 0 libva info: User requested driver 'iHD' libva info: Trying to open /opt/intel/mediasdk/lib64/iHD_drv_video.so libva info: Found init function __vaDriverInit_0_32 libva info: va_openDriver() returns 0 Pipeline surfaces number (DecPool): 20 MFX HARDWARE Session 0 API ver 1.23 parameters: Input video: AVC Output video: AVC Session 0 was NOT joined with other sessions Transcoding started .. Transcoding finished Common transcoding time is 0.094794 sec ------------------------------------------------------------------------------- *** session 0 PASSED (MFX_ERR_NONE) 0.094654 sec, 101 frames -i::h264 test_stream.264 -o::h264 out.264 ------------------------------------------------------------------------------- The test PASSED
The sample code is modified base on our original tutorial code, simple_decode and simple_encode. The call to the FFMpeg integration is added to the original source code, the modified area is wrapped by the following comment line:
// =========== ffmpeg splitter integration ============ ...... // =========== ffmpeg splitter integration end ============
The structure demuxControl keeps the control parameters of the demux process; the function openDemuxControl() initializes and configures the demuxControl structure; the structure is then used for the demux and decoding process; during the decoding, the function ffmpegReadFrame() reads the video frame after demuxing; finally the function closeDemuxControl() releases the system resources.
In the code "DECODE_AUDIO" turns on the audio decoding and demux the audio stream and use the FFMpeg audio decoder to uncompress the audio stream into the raw audio file "Audio.dat".
The structure muxControl keeps the control parameters of the mux process; the function openMuxControl initializes and configures the muxControl structure, the structure is then used for the encoding and mux process; during the encoding, the function ffmpegWriteFrame() writes the encoded stream into the output container via the FFmpeg muxer; finally the function closeMuxControl() releases the system resources.
In the code "ENCODE_AUDIO" turns on the audio encoding and mux/compress the audio raw data from "Audio.dat" to the video container.