Using MPEG-4 Decoder

Using MPEG-4 Decoder

sdhays@neon.com.tw's picture

I'm trying to build my own application based on the UMC framework from the IPP Media Samples, v5.0.017. I can play AVI and MP4 files using simple_player, and MJPEG AVIs play fine with the UMC example (from the UMC manual). But when I change the UMC example to use MPEG4VideoDecoder instead of MJPEGVideoDecoder, the program runs, seeming to process every frame, however not one frame is decoded for rendering. I'm not surprised that the MPEG4 decoder needs to be treated differently from the MJPEG decoder, but after searching for the last 3 days through simple_player.cpp, avsync.cpp, codec_pipeline.cpp, and umc_mpeg4_video_decoder.cpp, I'm still at a loss as to what I should do to get the MPEG4 decoder to work.

Any suggestions would be greatly appreciated!
Scott


P.S. As further note for when Intel updates the documentation: FIOReader is not built by default on Linux for IA32. The UMC example depends on FIOReader and thus cannot be built without modification. Changing FIOReader to FileReader works, but I don't know what difference (if any) this makes.

Message Edited by sdhays@neon.com.tw on 01-19-2006 06:35 PM

44 posts / 0 new
Last post
For more complete information about compiler optimizations, see our Optimization Notice.
rhy's picture

Hi Scott,



I am currently trying to decode the mpeg4 byte stream that comes directly from the GetFrame() method of the ipp mpeg-4 encoder, so we might look for the same thing.
Currently I am on the verge of dissecting codec_pipeline to find out how to feed the decoder correctly so it produces complete images.

However, when I feed the MPEG-4 decoder with what I get from the encoder, the decoder just produces some bytes of unusable data.



I guess what plays into it, is the way data comes in to the decoder. I am currently thinking this way:
* The MPEG-4 Stream does not consist of complete images side-by-side like the MJPEG stream does.

* This means that the MPEG-4 decoder needs some 'brain' containing the last image to produce the next image with the new bits of information coming in.

* Either this 'brain' is in the decoder itself, or it must be in the pipeline.


One thing I will try now for testing is to feed only I-Frames to the decoder, practically emulating MJPEG with that.


Rhy

Message Edited by Rhy on 01-20-2006 01:49 AM

vladimir-dudnik (Intel)'s picture

Hi Scott,


well, seems we have documentation issue here, it need to be addressed to the next release.


Thanks,
Vladimir


sdhays@neon.com.tw's picture

Hi, Rhy. Did you ever get the MPEG-4 decoder to work separate from CodecPipeline?

Scott

vladimir-dudnik (Intel)'s picture

Hi Scott,


as a variant, I can suggest you to take a look on umc-manual.pdf file, chapter 3, Base Classess, Example of UMC Base Classes Usage. There is a simple example of Motion JPEG decoder from AVI to YUV file. It should not be too difficult to replace MJPEG related staff with MPEG4 staff. Just formal approach should work, replace included header files and change object names and you should get very simple MPEG4 decoder.


Regards,
Vladimir


sdhays@neon.com.tw's picture

That's exactly what I did, but it didn't work which is why I posted in the first place. It's my understanding that Rhy was having trouble getting it to work as well. So, I'm wondering what other steps are necessary for the MPEG4 decoder.

Scott

vladimir-dudnik (Intel)'s picture


Hi Scott,


Sorry, I forget to mention, in case of MPEG4 you need to replace YUY2 to YV12 in decoder initialization. Wetested this and it is work. Let me know if I need to upload modified sample here


Regards,
Vladimir

sdhays@neon.com.tw's picture

Thanks, Vladimir. I tried your suggestion, changing VDecPrams.cformat to UMC::YV12. This still just gives me an empty output file. Could you please attach your working example so that I can see what I'm doing wrong?

Thanks again for your help!
Scott

vladimir-dudnik (Intel)'s picture

Scott,


you changed code on


page 3-99, line 18


VDecParams.cformat = UMC::YUY2; -> VDecParams.cformat = UMC::YV12;


and this does not work for MPEG4VideoDecoder?
Ok, I'll attach sample in day or two.

Regards,
Vladimir
vladimir-dudnik (Intel)'s picture


Please find modified umc_doc_example in attached archive. In this sample all references to MJPEG were changed to MPEG4 and initialization was corrected for MPEG4 decoder. Please try and we will be glad to get any feedback from you regarding UMC media sample and MPEG4 decoder

Regards,
Vladimir

Message Edited by vdudnik on 02-10-2006 11:07 AM

sdhays@neon.com.tw's picture

Thank you for posting your sample, but unfortunately it produces the same results as my own update to the UMC example. It seems to decode, but it simply doesn't save anything to output.yuv. I've compiled this on Windows and Linux (on Linux, I changed FIOReader calls to FileReader calls). The results were the same. I used AVI files with MPEG4 data from different encoders, including one generated by video_enc_con. All of the AVI files I tested work with simple_player but not with the sample program. Very strange.

I did have to make some changes to the Visual Studio project in order to get this sample to build. But that shouldn't make a difference since I just added include lines to the compiler command and library paths to the linker command. I didn't touch the source code except for when I tried it on Linux (as mentioned above).

If you have any other suggestions, please let me know. Thanks again for helping me out with this!

Scott

Message Edited by sdhays@neon.com.tw on 02-15-2006 04:32 PM

jordimitja@telefonica.net's picture

I tested also the umc_doc_examples.zip in Windows XP but my problem is that this sample crash in the first GetNextVideoData():

// decoding & rendering loop (frame by frame)
while(umcRes == UMC::UMC_OK || umcRes == UMC::UMC_NOT_ENOUGH_DATA)
{
umcRes = avi_spl.GetNextVideoData(&in); -> crash !!!
if (umcRes != UMC::UMC_OK)
break;

The error returned in release version is:
runtime error R6016
- not enough space for thread data

I tried to decode an avi generated with XVID or DIVX codec and I dont known if this is the problem, Can I decode videos encoded with these codecs with the IPP MPEG-4 decoder?

Could you help me, please ?

Thanks you very much in advance !!

PS: I attach you the video that Im using in my tests

Jordi.

jordimitja@telefonica.net's picture

I solved the problem, Im using visual c++ 6 and I see that I forgot in the project setting ->code generation-> use runtime library, select multithreaded DLL insteadof single threaded.


Im sorry if I spent your time.



sdhays@neon.com.tw's picture

So when you decode the file you posted, you get a file "output.yuv" that has your decoded video? No matter what file I try this with (including the one you posted), decoding SEEMS to work, but I always just end up with an empty (size 0) output.yuv.

Scott

jordimitja@telefonica.net's picture

I obtain the same result that you say in your mail a output.yuv file with size 0. I tried to get the output data with out.GetDataPointer() but I see that this pointer is NULL. With the sample player it works correctly but not with all the MPEG-4 avi files that I tested, I have any problems getting the avi information from some avi files. When I run the sample player shows:

Stream Type : AVI
Video Info :
-Video Type : UNDEF
-Resolution : 0x0
-Frame Rate : 0.00
Audio Info :
-Audio Type : UNDEF
-S.Frequency : 0
-Num.Channel : 0
-BitPerSample: 0

Could you test it please and tell me what is it wrong in this file? Is the avifile header wrong ?

I attached a bad avifile.

sdhays@neon.com.tw's picture

I'm sorry that it's not working for you, but I'm glad to hear that I'm not the only one having trouble getting this sample to work. ;-) I tried your AVI. I get the same result, but remuxing it with ffmpeg let it play okay, albeit a bit fast. So the problem, whatever it is, is in you AVI header.

Scott

jordimitja@telefonica.net's picture

Ok, Thank you very much for your test. I see also that If I reencode this avi using the Virtualdub the generated avi works perfectly. The problem its that I have a lot of avis (I generated this avis using the project avifile lib from sourceforge) and I tested these avis with a lot of players and I never had any problems. I would like to known what is it wrong in the header? You can test this avi with any video player and it works perfectly.

Jordi.

walkdeath@gmail.com's picture

I've the same problem. Is there something wrong with ipp5.0?

walkdeath@gmail.com's picture

But strangly, the simpleplayer can play mpeg4-encoded avi file well... I don't know why...

sdhays@neon.com.tw's picture

Vladimir, can you confirm that this sample works for you on MPEG4 AVI's? None of us seem to be able to get it to work. From my debugging, it seems that it's the MPEG4 decoder that's not producing anything, not any other part of the tool chain, but I still don't see why. This is much more strange if it works for you but not for us.

Thanks,
Scott

vladimir-dudnik (Intel)'s picture

Hi Scott,

I'm sorry, was busy last days, I watched discussion but had no time to answer. Yes, certainly, we tested this sample on several AVI files with MPEG4 content. The issue can be related with limitations in AVI splitter, it will be improved in new version, IPP v5.1, which is coming soon. Could you please attach some AVI file which you can't decode with the sample? So, we can test and probably find workaround before next release.

Regards,
Vladimir

vladimir-dudnik (Intel)'s picture

Ops, sorry, I was able to reproduce issue (just mistakenly used latest sources).


Well, there is workaround for that issue, you need to call VideoData.SetVideoParameters() function before VideoRender.RenderFrame(). I've attached MSVC project for your convenience, please let me know if it work for you.


Note, you need to unzip umc_mpeg4_dec folder to UMCapplications folder. And you need to pre-build UMC static libraries as this project depends from prebuild libraries for similicity.


Regards,
Vladimir


walkdeath@gmail.com's picture

hi~~~Thanks for your sample.
I've tried in linux(Fedora Core 4), using build32.sh to build the sample. But a "segment fault" problem came...
I found that the problem is in
if(umcRes == UMC::UMC_OK)

{

out.SetVideoParameters(

spl_info.m_video_info.clip_info.width,

spl_info.m_video_info.clip_info.height,

UMC::YV12);



umcRes = dst.UnLockInputBuffer(&out);

if (umcRes != UMC::UMC_OK)

break;



double time = dst.GetRenderFrame();



umcRes = dst.RenderFrame();

if (umcRes != UMC::UMC_OK)

break;

}
When I made "umcRes = dst.RenderFrame" a comment with "//".
Then there was no "segment fault" appeared, but it could only process 2 frames and also left me an empty "output.yuv" file.
And when I mad " out.SetVideoParameters..." a comment, it could process 11 frames and still, an empty "output.yuv" file.
Is there something wrong with the avi file that I process?
My avi file is 10M, so I can't attach it easily. Could anybody attach a small avi file?
Any suggestion will be appreciated!

walkdeath@gmail.com's picture

I've just done some tested and found something intereting.
In function "DecodeMPEG4Video", I add some code to test if the media data is correct.

// decoding & rendering loop (frame by frame)

while(umcRes == UMC::UMC_OK || umcRes == UMC::UMC_NOT_ENOUGH_DATA)

{

umcRes = avi_spl.GetNextVideoData(&in);


if (umcRes != UMC::UMC_OK)

break;

// printf("%d
", in.GetDataSize());
//this test prove that there is data in "in" whose type is MediaData ;

but after the follows,
umcRes = mpeg4_dec.GetFrame(&in, &out);
printf("in:%d
", in.GetDataSize());
printf("out:%d
", out.GetDataSize());

if (umcRes != UMC::UMC_OK && umcRes != UMC::UMC_NOT_ENOUGH_DATA)

break;

I only get "in:1 out:0". I think this proves that there is something wrong with GetFrame. But strangely, the "umcRes" passed the "if" test!
That really made me confused...

sdhays@neon.com.tw's picture

Thanks, Vladimir. Adding an out.SetVideoParameters() call fixed it for me on both Windows and Linux.

On a different, but related, topic: do you have a suggestion for how to play the output.yuv file? Can simpleplayer play it? If so, I haven't been able to get it to accept the output.yuv file. I've been playing it with YUV Viewer, a DirectShow plugin, but I was wondering if there was a better way. I haven't yet found the right command-line incantation to get MPlayer/FFmpeg to play it.

Thanks again for your help.
Scott

walkdeath@gmail.com's picture

Hi, scott
I've tried Vladimir's code in Fedora Core 4, but problems just come as I said.
Didn't you meet any problems when run the decoder in linux? There is always a "segment fault" when I run it... Could you offer some suggestion?

Thanks,
Wu Yu

sdhays@neon.com.tw's picture

Hi Wu Yu.
What file reader are you using? I had to change it to use the regular UMC::FileReader class for Linux. FIOReader doesn't even get built for Linux. I posted this issue earlier in another thread. Hopefully the next sample release will have a UMC sample that can work without any changes for both Windows and Linux.

#ifdef WIN32
UMC::FIOReader src;
#else
UMC::FileReader src;
#endif

That works for me on Red Hat 9.

Scott

walkdeath@gmail.com's picture

Hi Scott,
I've changed the file reader to FileReader, but still the same problem, "segment fault". There is nothing in the "VideoData out".
Could you please attach your sample code? Thanks very much!

Wu Yu

sdhays@neon.com.tw's picture

This is my edit of the file Vladimir originally posted. I've only added the setvideoparam() call that Vladimir added in his update (although my version only calls it once). This works for me on Red Hat 9.

Scott

walkdeath@gmail.com's picture

Hi Scott,
I've tried your code , but still the same problem, "segmentation fault". I've raise another issue "Problem about the FWVideoRender". If you have spare time, will you please have a look at it? I'm really confused about this problem... Thanks~~
Wu Yu

sdhays@neon.com.tw's picture

Some questions:
Have you tried this on many different AVIs made from different sources? Do those AVIs work flawlessly in simple_player? What compiler are you using? Are you dynamically or statically linking? Is there any way for you to try this on a different computer?

I've tried this on both a Windows XP system compiled with both the Intel Compiler 9.0 and the Microsoft compiler 7.1.xxx as well as gcc-3.4.5 on Red Hat 9. All of my builds use static linking, and they all work as expected. I haven't tried dynamic linking, so while it shouldn't be a problem, I just don't know.

From your description, it doesn't sound like the problem is in the renderer; it sounds like no frame is being decoded for the renderer to render and since it doesn't check for that possibility, it seg faults. That's what was happening earlier for me. So the root of the problem is probably farther up the pipeline; the AVI splitter seems most likely to me, especially since you've had some trouble with some of your AVI's with simple_player.

I've attached an AVI that works for me. It was encoded and multiplexed using a fairly recent CVS version of FFMpeg. If it still doesn't work for you, then I guess the next thing I'd want to try is building on a different computer if that's possible.

Good luck,
Scott

walkdeath@gmail.com's picture

Hi Scott,
Thanks very much for your spending time on my problem~~I've tried your avi file, "segmentation fault" again.
I've tried 3 avi files on two computer, all "segmentation fault".
But all of them run well using simple player, but it played somewhat fast when using your avi file. I have not learnt much about "make" and "link mode", so I use the "build32.sh" and "Makefile.lnx" supplied by the ipp_sample. I think I use gcc 4.0 and "static linkage mode", but I'm not very sure~~~
Shall we have a chat on the net if your have spare time? My MSN is kilua_wuyu@hotmail.com or we can use another way.
Thank you very much~~

Wu Yu

sdhays@neon.com.tw's picture

It runs faster in simple_player because it has no audio, so it plays as fast as it decodes. Specifying "-N" on the commandline should make it play at normal speed.

The default build for the IPP samples uses shared linking. Use ldd to find out if your binary has dependencies on the IPP (core) libraries if you're not sure if you're using static linking or or dynamic linking. I think I made a post mentioning some minor modifications for making the static linking work properly for simple _player using GCC earlier on this forum.

GCC 4.0 could possibly be a problem. It's not officially supported with the current version of the IPP libraries. GCC 3.4.5 is the latest supported version, I believe, so you may want to try that. Or you could download a trial version of the Intel compiler to see if that solves the problem.

Feel free to email me. I don't have a lot of time (and I'm running out of ideas for what might be wrong), but if I can help, I will.

Scott

vladimir-dudnik (Intel)'s picture

Hi Scott,


no, simple_player can't play raw YUV file. It is not difficult to write such utility (does not matter with using UMC or not) or you can look for some third-party YUV viewer, I was able to find several in google


Regards,
Vladimir


klend's picture
vdudnik wrote:

I've attached MSVC project for your convenience, please let me know if it work for you.

Note, you need to unzip umc_mpeg4_dec folder to UMCapplications folder. And you need to pre-build UMC static libraries as this project depends from prebuild libraries for similicity.

Regards,

Vladimir












Hi Vladimir,

Thanks for this example, together with the SetVideoParameters call from a later post I got this to work rather nicely. however, I do have a question regarding the call to mpeg4_dec.GetFrame(&in, &out):



What if I want to access the YV12 data in the 'out' buffer (instead of
just rendering them with FWVideoRenderer)?
I do not seem to find
a nice interface to the YUV parts except the
out.m_lpDest[0] .. out.m_lpDest[2] pointers.

Do I overlook the obvious? I did look at the NullVideoRenderer and thought about changing that a bit, but before I do, I want to know if I'm not missing something, as I think I do.



Please enlighten me,

Thanks in advance,

Hendrik

Message Edited by klend on 03-14-2006 12:55 PM

vladimir-dudnik (Intel)'s picture

Hi Hendrik

inthat version there is no methods to access to VideoData's internal data members, so you can add such or use direct access.

Regards,
Vladimir

jordimitja@telefonica.net's picture

Hi everybody:

I tested the wrong avi (bad_video.zip) with the new version IPP v5.1 and the results are the same. The simple_player fails and I cant reproduce this avi. However I solved the problem, its a problem in the avifile header, its wrong in the chunk "vids", I missed the fourcc handle necessary to decompress the video stream. Now it works correctly.

Thanks you very much !!!

Another question, Im using the MPEG-4 Decoder sample and I would like to use RGB24 output format but I see that the images are flipped. To fix this and obtain the images correctly I need to call to the function ippiMirror_8u_C3R after decompress the image:

IppStatus status = ippStsOk;
Ipp8u* pSrc[1];
pSrc[0] = (unsigned char*) out_.GetBufferPointer();
int srcStep[1];
srcStep[0] = pitch_;
IppiSize roiSize[1];
roiSize[0].width = image_width_;
roiSize[0].height = image_height_;

Ipp8u* pDst_mirror[1];
pDst_mirror[0] = ret.buffer.buffer();
IppiAxis flip_axis = ippAxsHorizontal;

status = ippiMirror_8u_C3R(pSrc[0], srcStep[0], pDst_mirror[0], srcStep[0],roiSize[0], (IppiAxis) flip_axis);

I would like to known if is it possible to configure any decoder option or any flags to avoid this operation? Im very worried about the performance impact because I need to decompress a lot of stream (up to 16). I need to use RGB24 because sometimes I must to realize any post-processing in decoded image, I known that the fastest decoder output format is YV12.

Thank you very much in advance !!

Jordi.

vladimir-dudnik (Intel)'s picture


Hello Jordi,


how did you get flipped image? UMC decoder does not rotate video frame!


Regards,
Vladimir

jordimitja@telefonica.net's picture

Hi Vladimir,

The frames that Im decoding are generated compressing the frames with XVID codec o DivX Pro Codec from a video capture card based in chip Bt878. If I store in BMP file format or I display a decompressed image in RGB24 format captured I can see the image correctly. If I display the frames decoded using XVID decoder or DivX decoder the images are not flipped.

Any ideas ?

Thank you very much in advance !!

Regards,

Jordi.

vladimir-dudnik (Intel)'s picture

Remember, that BMP support both top-left and bottom-left image orientation. You control this by changing sign for BITMAPINFOHEADER biHeight variable. Positive valuse means bottom-left orientation and so image will be flipped by Win32 API draw function. You need to set it to negative value to see image not flipped.
Note, encoders (as far as I know, any MPEG like) support only top-left image orientation.

Vladimir

jordimitja@telefonica.net's picture

Hi everybody,

I see where its the problem with the decoder. The problem its in the color conversion routines IPP from RGB to YUV. I read in a web page the next sentence:

"Color conversion from YUV 420 to RGB results in an upside-down image:
In some imaging pipelines, the interpretations of YUV images and RGB images are flipped relative to each other. The Intel IPP YUVToRGB function does not flip the image, but provides routines to perform the flip operation. One solution is to apply the function ippiMirror to flip the result before or after calling the YUVToRGB color conversion function. "

If Im using YUY2 from YV12 it works correctly and the image isnt inverted.

Regards,

Jorge.

vladimir-dudnik (Intel)'s picture


Hi Jorge,


if you use Windows BITMAP structure to represent/display decoded frame you can use negative height to allow Win32 API make flip for you during rendering


Regards,
Vladimir

henrik@zimmer.to's picture

Hi,

IPP seems great and we wanted to use it to speed up image processing/CV-related tasks on video streams.

As a first learning step i wanted to access the images decoded in the umc sample posted in this thread, and for example perform some filtering on each frame using IPPi.

How could i extract a frame (using e.g. the umc sample) into an appropriate Ippi container?

My problem could be related to the fact that i havn't managed to find out how the "in/out" "MediaData/VideoData" work, what they contain and thus how to extract frame information (in some format) into a IPPImage structure!?

Thankful for all replies, samples and reading references!









(For some first feedback:

i think the downloadable samples are sometimes (perhaps necessarily) to big, and it is difficult to find a quick solution to a small limited problem.

The UMC manual and the samples could also be a bit more elaborat for the "newbie".

However on this forum i have found very good advice and many helpful people!)

vladimir-dudnik (Intel)'s picture


Thanks for your valuable feedback, we add us energy to improve UMC documenation and IPP samples, I hope,they will be noticeable better in the next version (IPP v6.0)


Regards,
Vladimir

Login to leave a comment.