H264 decoder latency by TopoEdit

Lily D.

qua, 14/11/2012 - 03:54

I have downloaded the latest Intel SDK (Media SDK 2012 R3), and registered the mfx_mft_h264vd_32.dll to my computer (Win8, 9200, 32bit).

By using the TopoEdit (Media Foundation tool), I add the video capture source, add the Intel Media SDK H.264 Decoder MFT, add the EVR, and then preview. I find that it previewed successfully but with some latency.

I debugged the source code(sample_mfoundation_plugins) attaching to the TopoEdit.exe. According to the trace information, at the beginning, the decoder would buffer some samples before decoding and calling ProcessOutput to output sample.

Later, I set

mfxVideoParam::AsyncDepth = 1 and

mfxBitStream::DataFlag = MFX_BITSTREAM_COMPLETE_FRAME

The trace information show that the decoder would buffer 5 samples before decoding. The format of the video from camera capture is H264, 1280*720, 10Fps.

However I want to preview the H264 real-time without latency.

Who can tell me how to implement the real-time preview H264?

Or which parameters should be set to control that the decoder will not buffer samples before decoding.

Thanks!

9 posts / 0 new

Último post

Para obter mais informações sobre otimizações de compiladores, consulte Aviso sobre otimizações.

camkego

qua, 14/11/2012 - 18:03

Lily,

This is an interesting question, I am a developer working with the Media SDK too. Although I have done little with Media Foundation and the Media SDK, I have done some work with DirectShow and the Media SDK.

If you are indeed working with a USB H.264 camera under Media Foundation,
I think you will find fair bit of latency even with little latency in the H.264 decoder.
I also believe the rendering filter at the end of your graph may add a fair bit of latency in order to have a playback buffer
capable of smoothing out jitter in the frame arrivals, you may want to look into this also.

Are you able to share your Topo graph?

Cameron

Lily D.

qua, 14/11/2012 - 18:49

Dear Cameron:

The Topo graph just like this.

I also test the latency with the format H264, 640*480, 30Fps. The trace information shows that the decoder would buffer 16 samples before decoding.

The latency of 1280*720, 10Fps is about 720ms. Today the latency of 640*480 30Fps has not been tested due to some reason, but in my opinion, it may 700ms~800ms.

So, since the fair bit of latency is not avoided, I just want to know how to make the decoder not buffer samples or buffer one sample only to reduce the latency as far as possible.

Thank you
Lily.

Цитата:

camkego wrote:
Lily,

This is an interesting question, I am a developer working with the Media SDK too. Although I have done little with Media Foundation and the Media SDK, I have done some work with DirectShow and the Media SDK.

If you are indeed working with a USB H.264 camera under Media Foundation,
I think you will find fair bit of latency even with little latency in the H.264 decoder.
I also believe the rendering filter at the end of your graph may add a fair bit of latency in order to have a playback buffer
capable of smoothing out jitter in the frame arrivals, you may want to look into this also.

Are you able to share your Topo graph?

Cameron

Lily D.

qua, 14/11/2012 - 18:51

Topo Graph

Anexos:

Anexo	Tamanho
Download topology.jpg	17.47 KB

camkego

qua, 14/11/2012 - 19:42

I suspect you will have a hard time reducing the latency much:
- you are using a USB Camera probably will little concern for low-latency design
- the camera most likely cannot generate an Intel MSDK conforming low-latency AVC stream
- probably the Media Foundation renderer does buffering to ensure it can deliver frames at the right time
- there may be other buffer points in the MF graph which you have little or no control of

I wouldn't be suprised if you figure out how to reduce some latency, it just might be hard or impossible to break 300-250ms,
at least this has been my experience with 3rd party filters and DirectShow.

Please let us know here about your progress, I only have suspicions, not enough hard experience with MF to say for sure.
Hopefully I am wrong.

Regards,
Cameron

Petter Larsson (Intel)

qui, 15/11/2012 - 12:19

Hi,

Regarding configuring Intel Media SDK components for low latency and to enable other features related to video conferencing, please refer to the following white paper:
http://software.intel.com/en-us/articles/video-conferencing-features-of-...

To achieve low latency decode using Media SDK you are also impacted by the format of the input stream. For instance, if the input stream uses B-frames then additional decoder latency will naturally occur.

Regards,
Petter

Lily D.

qui, 15/11/2012 - 17:13

Hi Petter:

In my test, the input stream use 9 B-frames between two I-frame. And then I made the input stream using 2 B-frames between two I-frame or using totally I-frames, but the latency do not improve. The trace information also show that the decoder will buffer 5 samples before decoding.
Is there an internal buffer set to cache data in the decoder? If it has, how can i change the internal buffer size?

Best Regards

Lily

Цитата:

Petter Larsson (Intel) wrote:
Hi,

Regarding configuring Intel Media SDK components for low latency and to enable other features related to video conferencing, please refer to the following white paper:
http://software.intel.com/en-us/articles/video-conferencing-features-of-...

To achieve low latency decode using Media SDK you are also impacted by the format of the input stream. For instance, if the input stream uses B-frames then additional decoder latency will naturally occur.

Regards,
Petter

Petter Larsson (Intel)

sex, 16/11/2012 - 15:57

Hi Lily,

if you look at the referenced white paper you will find information on how to encode your stream so that decoder can decode with low latency.
Having no B-frames is essential, but also make sure that decoder picture buffer (DPD) gets set to 1 as described in the paper.

Regards,
Petter

Lily D.

seg, 07/01/2013 - 00:14

I have resloved the problem by using DecodeFrameAsync(NULL,..) to pull the cached sample.
Thanks for everyone's help.

Voltar à postagem original

Intel® Developer Zone

H264 decoder latency by TopoEdit

H264 decoder latency by TopoEdit

Anexos:

Faça login para deixar um comentário.