The application can use the SDK encoding, decoding and video processing functions together for transcoding operations. This section describes the key aspects of connecting two or more SDK functions together.
The application passes the output of an upstream SDK function to the input of the downstream SDK function to construct an asynchronous pipeline. Such pipeline construction is done at runtime and can be dynamically changed, as illustrated in Example 5.
Example 5: Pseudo Code of Asynchronous Pipeline Construction
The SDK simplifies the requirement for asynchronous pipeline synchronization. The application only needs to synchronize after the last SDK function. Explicit synchronization of intermediate results is not required and in fact can slow performance.
The SDK tracks the dynamic pipeline construction and verifies dependency on input and output parameters to ensure the execution order of the pipeline function. In Example 5, the SDK will ensure MFXVideoENCODE_EncodeFrameAsync does not begin its operation until MFXVideoDECODE_DecodeFrameAsync or MFXVideoVPP_RunFrameVPPAsync has finished.
During the execution of an asynchronous pipeline, the application must consider the input data in use and must not change it until the execution has completed. The application must also consider output data unavailable until the execution has finished. In addition, for encoders, the application must consider extended and payload buffers in use while the input surface is locked.
The SDK checks dependencies by comparing the input and output parameters of each SDK function in the pipeline. Do not modify the contents of input and output parameters before the previous asynchronous operation finishes. Doing so will break the dependency check and can result in undefined behavior. An exception occurs when the input and output parameters are structures, in which case overwriting fields in the structures is allowed. (Note that the dependency check works on the pointers to the structures only.)
There are two exceptions with respect to intermediate synchronization:
1. The application must synchronize any input before calling the SDK function MFXVideoDecode_DecodeFrameAsync, if the input is from any asynchronous operation.
2. When the application calls an asynchronous function to generate an output surface in video memory and passes that surface to a non-SDK component, it must explicitly synchronize the operation before passing the surface to the non-SDK component.
When connecting SDK function A to SDK function B, the application must take into account the needs of both functions to calculate the number of frame surfaces in the surface pool. Typically, the application can use the formula Na+Nb, where Na is the frame surface needs from SDK function A output, and Nb is the frame surface needs from SDK function B input.
For performance considerations, the application must submit multiple operations and delays synchronization as much as possible, which gives the SDK flexibility to organize internal pipelining. For example, the operation sequence, ENCODE(f1) à ENCODE(f2) àSYNC(f1) à SYNC(f2) is recommended, compared with ENCODE(f1) àSYNC(f1) àENCODE(f2) àSYNC(f2).
In this case, the surface pool needs additional surfaces to take into account multiple asynchronous operations before synchronization. The application can use the AsyncDepth parameter of the mfxVideoParam structure to inform an SDK function that how many asynchronous operations the application plans to perform before synchronization. The corresponding SDK QueryIOSurf function will reflect such consideration in the NumFrameSuggested value. Example 6 shows a way of calculating the surface needs based on NumFrameSuggested values.
Example 6: Calculate Surface Pool Size
During asynchronous pipeline construction, each stage SDK function will return a synchronization point (sync point). These synchronization points are useful in tracking errors during the asynchronous pipeline operation.
Assume the pipeline is AàBàC. The application synchronizes on sync point C. If the error occurs in SDK function C, then the synchronization returns the exact error code. If the error occurs before SDK function C, then the synchronization returns MFX_ERR_ABORTED. The application can then try to synchronize on sync point B. Similarly, if the error occurs in SDK function B, the synchronization returns the exact error code, or else MFX_ERR_ABORTED. Same logic applies if the error occurs in SDK function A.