Adding and Retrieving Closed-caption messages in AVC and MPEG-2 Streams

By Sravanthi Kota Venkata, Published: 08/18/2014, Last Updated: 08/18/2014

Legal Disclaimer

In this article, we are going to show how to add and retrieve closed-caption data to the AVC and MPEG2 streams through Intel® Media Server Studio and Intel® INDE. In the first part, we will illustrate using code examples how to add CC messages to the AVC and MPEG2 streams. In the second part, we will show how to retrieve these messages using the decoder.

To test the closed-captioning in AVC/MPEG2 using the following code snippets, we recommend using the tutorials instead of the samples. (You can download the tutorials from the TUTORIALS tab here). The tutorials are much easier to understand and have more comments to help you. For encoding, simple_6_encode_vmem_lowlatency tutorial, and for decoding, simple_2_decode tutorial.  In the code snippets below, we point you to the location where this code should be added for encode and decode.

Let's get started!

Adding Closed-caption Messages to the AVC and MPEG2 Encode Stream

In this section, we will see how to add closed-caption data to AVC and MPEG2 streams during the encoding stage. Below are the high-level how-to steps, and we will follow that up with a code snippet.

Step 1: Create the SEI/user_data payload with the appropriate header and message fields. For details on the AVC and MPEG2 payload structures, refer to this document. You can also refer to the following -  for AVC stream (Section 6.4.2 in the document), and for MPEG2 stream (Section 6.2.3 in the document).

Step 2a: Populate the mfxPayload structure with this payload and header information.

Step 2b: Pass the mfxPayload structure to mfxEncodeCtrl structure 

Step 3: Pass this structure to the EncodeFrameAsync function (first parameter), and you're done!

Code example to add CC messages to AVC, in the main encoding loop:

#define MESSAGE_SIZE 20
typedef struct
				 unsigned char countryCode;
				 unsigned char countryCodeExtension[2];
				 unsigned char user_identifier[4];
				 unsigned char type_code;
				 unsigned char payloadBytes[MESSAGE_SIZE];
				 unsigned char marker_bits;
			} userdata_reg_t35;
			/*** STEP 1: START: SEI payload: refer to <link to AVC> for the format */
			/* Populating the header for the SEI payload. In most cases, these assignments will not change */
			userdata_reg_t35 m_userSEIData;
			m_userSEIData.countryCode = 0xB5;
			m_userSEIData.countryCodeExtension[0] = 0x31;
			m_userSEIData.countryCodeExtension[1] = 0x00;
			m_userSEIData.user_identifier[0] = 0x34;
			m_userSEIData.user_identifier[1] = 0x39;
			m_userSEIData.user_identifier[2] = 0x41;
			m_userSEIData.user_identifier[3] = 0x47;
			m_userSEIData.type_code = 0x03;
			m_userSEIData.marker_bits = 0xFF;
			/* Populate the actual message. In this example it is "Frame: <frameNum>" */
			sprintf((char*)m_userSEIData.payloadBytes, "%s%d", "Frame: ", nFrame);
			/*** STEP 1: END: SEI payload: refer to <link to AVC> for the format */

			/*** STEP 2a: START: Fill mfxPayload structure with SEI Payload */
			mfxU8 m_seiData[100]; // Arbitrary size
			mfxPayload m_mySEIPayload;
			memset(&m_mySEIPayload, 0, sizeof(m_mySEIPayload));
			m_mySEIPayload.BufSize = sizeof(userdata_reg_t35) + 2; // 2 bytes for header
			m_mySEIPayload.NumBit = m_mySEIPayload.BufSize * 8;
			m_mySEIPayload.Data = m_seiData;

			// Insert SEI header and SEI msg into data buffer
			m_seiData[0] = (mfxU8)m_mySEIPayload.Type; // SEI type
			m_seiData[1] = (mfxU8)(m_mySEIPayload.BufSize-2); // Size of following msg
			memcpy(m_seiData+2, &m_userSEIData, sizeof(userdata_reg_t35));
			mfxPayload* m_payloads[1]; 
			m_payloads[0] = &m_mySEIPayload;
			/*** STEP 2a: END: Fill mfxPayload structure with SEI Payload */

			/*** STEP 2b: START: Encode control structure initialization */
			mfxEncodeCtrl m_encodeCtrl;
			memset(&m_encodeCtrl, 0, sizeof(m_encodeCtrl));
			m_encodeCtrl.Payload = (mfxPayload**)&m_payloads[0];
			m_encodeCtrl.NumPayload = 1;
			/*** STEP 2b: END: Encode control structure initialization */
			nEncSurfIdx = Get Free Surface;
			Sucface Lock;
			pmfxSurfaces[nEncSurfIdx] = Load Raw Frame;
			Surface Unlock;
			/*** STEP 3: Encode frame: Pass mfxEncodeCtrl pointer */
			sts = mfxENC.EncodeFrameAsync(&m_encodeCtrl, pmfxSurfaces[nEncSurfIdx], &mfxBS, &syncp);


Code example to add CC messages to MPEG2, in the main encoding loop:

Adding payloads to MPEG2 is similar to the AVC example above. The difference comes from the format used for the MPEG2 user_start_code as compared to SEI message. Below, we illustrate how to populate an MPEG2 payload. This is in accordance to the ATSC standard for MPEG2 Video.

#define USER_START_CODE 0x1B2
#define MESSAGE_SIZE 20
typedef struct{
				mfxU8 atsc_identifier[4];
				mfxU8 type_code;
				/* For type 0x03, some additional bits before the data field starts - refer to <link to cc_data()> */
				mfxU8 additional_bits[2];
				unsigned char cc_data[MESSAGE_SIZE];

			/** STEP 1 */
			ATSC_user_data atsc;
			atsc.atsc_identifier[0] = (mfxU8)0x34;
			atsc.atsc_identifier[1] = (mfxU8)0x39;
			atsc.atsc_identifier[2] = (mfxU8)0x41;
			atsc.atsc_identifier[3] = (mfxU8)0x47;
			atsc.type_code = 0x03;					
			atsc.additional_bits[0] = (mfxU8)0x12;	//00010010;		//cc_count, addnl data, cc data, em data processed
			atsc.additional_bits[1] = (mfxU8)0xff;		//reserved			
			sprintf((char*)atsc.cc_data, "%s%d", "Frame: ", nFrame);
			/** STEP 2a */
			mfxU8 m_seiData[100]; // Arbitrary size
			mfxPayload m_Payload;
			memset(&m_Payload, 0, sizeof(m_Payload));
			m_Payload.Type = USER_START_CODE;
			m_Payload.BufSize = MESSAGE_SIZE + 7;
			m_Payload.NumBit = m_Payload.BufSize * 8;
			m_Payload.Data = m_seiData;
			memcpy(m_seiData, &atsc, 7);
			memcpy(m_seiData+7, &atsc.cc_data, MESSAGE_SIZE);
			mfxPayload* m_payloads[1]; 
			m_payloads[0] = &m_Payload;

			/** STEP 2b */
			mfxEncodeCtrl m_encodeCtrl;
			memset(&m_encodeCtrl, 0, sizeof(m_encodeCtrl));
			m_encodeCtrl.Payload = (mfxPayload**)&m_payloads[0];
			m_encodeCtrl.NumPayload = 1;
			nEncSurfIdx = Get Free Surface;
			Sucface Lock;
			pmfxSurfaces[nEncSurfIdx] = Load Raw Frame;
			Surface Unlock;
			/*** STEP 3 */
			sts = mfxENC.EncodeFrameAsync(&m_encodeCtrl, pmfxSurfaces[nEncSurfIdx], &mfxBS, &syncp);

In this section, we have seen how to add payloads to the AVC and MPEG2 encode stream. You can verify the payloads are added by opening the output bitstream in an editor and viewing in HEX mode (for instance, GVim). You should see each frame carrying a payload that has the "Frame: <framenum>" along with the payload meta information. Below is an encoded out.h264 file that uses the above-mentioned code snippet in simple_6_encode_vmem_lowlatency tutorial to add CC captions. You can see the SEI message "Frame: 6" highlighted.

Retrieving Messages from the Decoded Stream

We just saw how to add CC messages to the encode stream. In this section, we will see how to retrieve the encoded SEI/userData messages from the AVC or MPEG2 streams respectively. The SDK provides an API GetPayload() for this purpose, and we will illustrate how to use this for AVC and MPEG cases. The GetPayload() function follows the DecodeFrameAsync() function call, and returns the mfxPayload structure populated with the message, number of bytes and timestamp. Please note that you are required to initialize the BufSize and Data members of the structure before you call the GetPayload() function.

sts = mfxDEC.DecodeFrameAsync(&mfxBS, pmfxSurfaces[nIndex], &pmfxOutSurface, &syncp);

	mfxPayload dec_payload;
	mfxU64 ts;
	dec_payload.Data = new mfxU8[100];
	dec_payload.BufSize = MESSAGE_SIZE;
	dec_payload.NumBit = 1;

	/* Since the decode function is asynchronous, we will loop over the GetPayload() function until we drain all the messages ready */
	while(dec_payload.NumBit > 0)
		mfxStatus st = mfxDEC.GetPayload(&ts, &dec_payload);
		#ifdef AVC
		#ifdef MPEG2
				fwrite(dec_payload.Data, sizeof(unsigned char), dec_payload.BufSize, stdout); //For debug purpose - Prints out the payload on to the screen.

Below is the output of the fwrite for decode of out.h264 file we encoded above. On the console is printed the output of the GetPayload() call for each frame.

This concludes our howto add and retrieve CC messages in AVC and MPEG2 stream. You can also refer to our documentation section 4.14 for the pseudo-code, and for information on the APIs, you can refer to the mediasdk-man.pdf in the doc folder of the Media SDK install folder.

Product and Performance Information


Intel's compilers may or may not optimize to the same degree for non-Intel microprocessors for optimizations that are not unique to Intel microprocessors. These optimizations include SSE2, SSE3, and SSSE3 instruction sets and other optimizations. Intel does not guarantee the availability, functionality, or effectiveness of any optimization on microprocessors not manufactured by Intel. Microprocessor-dependent optimizations in this product are intended for use with Intel microprocessors. Certain optimizations not specific to Intel microarchitecture are reserved for Intel microprocessors. Please refer to the applicable product User and Reference Guides for more information regarding the specific instruction sets covered by this notice.

Notice revision #20110804