Decoder outputs the same timestamp twice?

Decoder outputs the same timestamp twice?

dr_asik's picture

Hi, following my previous question, I did figure out how to provide timestamps to the decoder, by setting the TimeStamp property of the mfxBitStream together with the new data on each frame. I'm running into a strange case where the decoder will output two frames with the same timestamp although I only provided one frame with that specific timestamp. I have a feeling I'm not using this property correctly.

Indeed here's basically how I'm feeding data to the decoder

// Move remaining data back to the buffer start

memmove(m_mfxBitStream->Data, m_mfxBitStream->Data + m_mfxBitStream->DataOffset, m_mfxBitStream->DataLength);

// Copy new data after existing data

memcpy(m_mfxBitStream->Data + m_mfxBitStream->DataLength, newDataPtr, newDataSize);

// Update bitstream fields

m_mfxBitStream->TimeStamp = newDataTimeStamp;

m_mfxBitStream->DataOffset = 0;

m_mfxBitStream->DataLength += newDataSize;

The potential issue I can see with this is that the new timestamp is not guaranteed to correspond to the new data in the bit stream, i.e. it's possible there is still remaining data from previous frame(s) and this timestamp will be associated with an older frame or even possibly several frames. Am I correct that this is the likely explanation for the doubled timestamp I'm seeing in the output, and how can I remedy this problem?

Thanks!

7 posts / 0 new
Last post
For more complete information about compiler optimizations, see our Optimization Notice.
Petter Larsson (Intel)'s picture

Hi,

Artificially inserting timestamps for a frame stored in mfxBitstream will not provide the desired results in most cases, unless your stream has only I frames or I frame followed by sequence of typical P frames.

The reason for this is that the encoded frames in the stream is not guaranteed to be stored in sequential order due to frame dependencies. Instead a common scenario is that the corresponding input timestamps are stored with the frame a conatiner(e.g. mp4, mkv etc.)

Not sure this answers your question, but it is likely related to the issue you facing.

Also, to ensure correct timestamp behavior, make sure that you only feed one frame at a time (in the mfxbitstream) to the decoder, and set DataFlag to MFX_BITSTREAM_COMPLETE_FRAME.

Regards,
Petter 

dr_asik's picture

Thanks for pointing out the MFX_BITSTREAM_COMPLETE_FRAME flag, that seems to improve things somewhat. Still, I'm running into the same issue after a while.

I'm aware of what B-frames are, and this is precisely the reason why we need the decoder to tell us which output frame corresponds to which input (otherwise we could just assume frames come out in the order we feed them). We do this with AvCodec by specifying the AvPacket::pts when sending data to the decoder, and looking at the AvFrame::pkt_pts field on the output frame. Nvidia's decoder works in a similar way. I'm pretty confident it's possible to get this to work with your decoder as well.

I investigated a bit more and I found that most of the time, after a call to DecodeAsync, the bitstream's DataLength is 0. All works well in this case. However, sometimes the DataLength is non-zero, and this is where the next frame we feed will have its timestamp appear twice in output frames. To illustrate:

bitstream is empty
feed frame with timestamp 30
decode -> output frame with timestamp 28
bitstream is empty
feed frame with timestamp 31
decode -> output frame with timestamp 29
bitstream NOT empty (DataLength != 0)
feed frame with timestamp 32
decode -> output frame with timestamp 30
bitstream is empty
feed frame with timestamp 33
decode-> output frame with timestamp 32
bitstream is empty
feed frame with timestamp 34
decode -> output frame with timestamp 32 <- oops got that one already!

So basically I guess my question is, what should I do in case the DataLength is not zero to ensure that the next timestamp will not overlap two frames?

andy4us's picture

If DataLength is not zero, then it seems as though you have more than one NAL block in the bitstream.

In sample_utils.cpp is a function, mfxStatus ReadNextFrame(mfxBitstream *pBS) which will populate a mfxBitstream based on NAL blocks, which is what the decoder wants.

Also, I would make an array of mfxBitStreams on a per frame basis and feed those in. Life becomes much less complicated then ! By reusing the same mfxBitstream and setting the timestamp, you're making assumptions about your frame numbering.

dr_asik's picture

I'm quite confused. The function you refer is I assume CH264FrameReader::ReadNextFrame(mfxBitStream* pBS), which reads a single H264 frame from a file on disk. In my application, frames come as memory buffers containing single encoded frames - the architecture is already in place to read and parse H264 data from disk or network or whatever. I can make assumptions about the frame numbering because the frames are numbered by code that understands H264. I'm trying to understand how to pass that numbering to the Intel decoder in a way that it understands "this number corresponds to this data". So far simply setting the timestamp on the bitstream doesn't do the trick because sometimes the decoder can output more than one frame, and the timestamp gets applied to both. 

andy4us's picture

Ok, just by going by what you had written to start with.

The first one that you labelled 32 is actually frame 31.

After feeding frame 31, and having data left, instead of then feeding the remaining data in the bitstream again as frame 31, you code will add frame 32 onto the remaining data for 31 and call it 32. The decoder consumes the remainder of 31, but it's been called 32.Thats why your first Frame 32 is actually Frame 31. You state you output frames are 28,29,30,32,32

The decoder should return a different error code ( I think ! ) in the instance when it does this ( DEVICE_BUSY ?), but check the documentation to be sure.

A simple while loop based on the return code and bitstream empty may fix you problem.

dr_asik's picture

Yes, I just read this in the documentation and reworked the code in consequence:

It is recommended that the application invoke [MFXVideoDECODE_DecodeFrameAsync] repeatedly until the function returns MFX_ERR_MORE_DATA, before appending any more data to the bitstream buffer.

I have an assert that DataLength is 0 every time this function returns and so far so good on every stream I tested it with. I think things are working fine now. Thanks!

Login to leave a comment.