Showing Different Results in Neural Compute Stick and CPU

Showing Different Results in Neural Compute Stick and CPU


Hi,

We have trained efficientnet model for classification. While I am testing the model (FP16 model) in NCS2 and CPU, the output labels shown are different in both the devices. FP32 and FP16 model execution in CPU shows the same results only. Could you please support us?

Looking forward for your reply.

Thank you,

Suchithra

5 posts / 0 new

Hi Suchithra,

Thank you for reaching out.
The results in FP16 and FP32 tested on CPU could be the same as they are using the same processor. On the other hand, if you compare the Myriad output to CPU it may differ slightly but classification results should be close. I have tried the demo_squeezenet demo for CPU (FP16 and FP32) vs Myriad (FP16) and the results are close enough (see below). May I ask how results differ on your end?

MYRIAD X FP16

Top 10 results:

Image <C:\openvino space\openvino\deployment_tools\demo\\car.png>

classid probability label

------- ----------- -----

817     0.6708984   sports car, sport car

479     0.1922607   car wheel

511     0.0936890   convertible

436     0.0216064   beach wagon, station wagon, wagon, estate car, beach waggon, station waggon, waggon

751     0.0075760   racer, race car, racing car

656     0.0049667   minivan

717     0.0027428   pickup, pickup truck

581     0.0019779   grille, radiator grille

468     0.0014219   cab, hack, taxi, taxicab

661     0.0008636   Model T

CPU FP16

Top 10 results:

Image <C:\openvino space\openvino\deployment_tools\demo\\car.png>

classid probability label

------- ----------- -----

817     0.6853030   sports car, sport car

479     0.1835197   car wheel

511     0.0917197   convertible

436     0.0200694   beach wagon, station wagon, wagon, estate car, beach waggon, station waggon, waggon

751     0.0069604   racer, race car, racing car

656     0.0044177   minivan

717     0.0024739   pickup, pickup truck

581     0.0017788   grille, radiator grille

468     0.0013083   cab, hack, taxi, taxicab

661     0.0007443   Model T

CPU FP32

Top 10 results:

Image <C:\openvino space\openvino\deployment_tools\demo\\car.png>

classid probability label

------- ----------- -----

817     0.6851521   sports car, sport car

479     0.1835010   car wheel

511     0.0918672   convertible

436     0.0200784   beach wagon, station wagon, wagon, estate car, beach waggon, station waggon, waggon

751     0.0069436   racer, race car, racing car

656     0.0044373   minivan

717     0.0024768   pickup, pickup truck

581     0.0017814   grille, radiator grille

468     0.0013093   cab, hack, taxi, taxicab

661     0.0007501   Model T

 

If you have more questions, feel free to post here again.

Regards,

David


We have trained model using PyTorch framework. We converted the trained .pth model into ONNX and then used Intel OpenVINO model optimizer for converting ONNX model to IR format.

Then deployed the IR model (FP16 model) on MYRIAD X VPU(NCS2) and CPU(Intel Pentium N42000, Core i7 and Core i5).

It has been found that our trained EfficientNetB4 model, outputs different result  on NCS2 and CPU. We would like to know why this happens? While we have tested different models on both the devices, we didn't find any difference.

Please find the table attached with sample of outputs that varies, for your reference.

 


Hi Suchithra,

Thank you for your reply. 

Unfortunately, there are no attached files, so the table you sent cannot be seen. Could you send it again?

Regarding your results, could you please provide the following for us to test on our end:

- The frozen model you are using and the model optimizer command used to convert the IR files.

- The program/sample you used to test.

- Expected input/output samples.

Best regards,

David


Hi,

I have a similar issue with a Resnetv2 model. Where the output from the tensorflow model does not match the NCS2 output sufficiently for some inputs. But when settings 'VPU_HW_STAGES_OPTIMIZATION': 'NO' both outputs are sufficiently close.

Is there a way to share the model non-publicly ?

Leave a Comment

Please sign in to add a comment. Not a member? Join today