When I use quicksync to encode to an h.264 file, then mux to mp4 using mp4box, Quicktime (Windows version) generally will not play the file correctly. Other players play it fine. The mux is apparently not the problem because the same mux command works on h.264 files from another h.264 encoder. I am encoding using a modified sample_encode from the media sdk examples. I am on a Core i7 3770k (HD 4000), Windows 7 Pro SP1 x64, with latest graphics driver (188.8.131.5261).
It doesn't matter whether I use hw or sw encode. I have tried baseline, main, and high profiles.
I can make it more or less work by setting m_mfxEncParams.mfx.GopPicSize = 1. However even in this case Quicktime comes up showing a green or black first frame, which it doesn't do normally, but at least in that case when it starts playing it shows good frames. When I set GopPicSize to higher numbers, Quicktime seems to show about that many green frames before showing good frames.
I can't believe that Quicktime doesn't support B frames. Is there some other parameter I am missing?