I'm currently using the h.264 decoder in an in-house app in two different scenarios. When I feed the decoder a stream created with the QuickSync encoder, I don't have any frame latency from when I submit my first frame to when an output frame is available. But if I decode a stream made with a different encoder, I have two frames of latency on the output side before I start getting output. This stream is a only I and P frames. With other decoders, I don't get any latency like this, so I'm assuming that I'm doing something wrong on my end.
I am setting the MFX_BITSTREAM_COMPLETE_FRAME flag in the DataFlag field for the mfxBitstream object and I've also set AsyncDepth = 1 in the decoder parameters. Is there anything else I should be looking for? Is the decoder looking for an AUD NAL to signal the end of a frame?