I've been building a video conferencing application based on the encoder & decoder in the SDK (namely, H.264 codec, using 3.0 beta 2).
The encoder works wonders, with extremely low latency, However, the decoder appears to have extremely high latency ! I know it is the decoder (decoding to system memory, by the way), because when I connect another decoder to the encoder's output, I receive almost no latency (however, CPU utilization suffers).
I've gone over all the tricks listed in the manual and the videoconf samples, including setting AsyncDepth to 1 (on both encoder & decoder), setting the MaxDecFrameBuffering to 1, GopRefDist to 1...
When encoding 640x480 30fps, I see a latency of about 1/2 second. What I've noticed was that when I call
on the decoder, I get a frame-count of 17, which almost exactly matches the perceived latency.
The only paramter I've changed which has any effect over it is the codec level, but this is tied to the resolution passed and framrate, which I can not change. I hardly understand why these parameters affect the latency so.
What I'm askin in short is : what am I missing here, in shortening the latency ? I doubt this latency is inherent to the decoder, which is otherwise very good !