Running first inference call is significantly slower

Running first inference call is significantly slower

Hi,

I am facing this peculiar behavior that I cannot solve. I am using opencv with the inference engine backend to do my inferencing. I have trained my own mobilnet V2 for inferencing, and converted it to IR. I am able to do the inferencing, however the first call to net.forward() is always significantly slower than the subsequent calls. For example in milliseconds:

6190 (first inference)

23 

22

22

23

22

23

24

24

24

24

25

 

I have initialised the network beforehand and am only measuring the tome for inference. Eg:

    Net net = readNetFromModelOptimizer(xml, bin);
    net.setPreferableBackend(DNN_BACKEND_INFERENCE_ENGINE);
    net.setPreferableTarget(DNN_TARGET_MYRIAD);

 

for loop across vector containing images to inference:

{

    Mat image, blob;
    blob = blobFromImage(inputImg);
    Mat prob;
    auto start = chrono::steady_clock::now();   

    net.setInput(blob);
    prob = net.forward();
    float probF = prob.at<float>(0);
    auto ending = chrono::steady_clock::now();
    cout << chrono::duration_cast<chrono::milliseconds>(ending - start).count() << endl;

}

 

Could anyone help me figure out what I'm doing wrong? Or is there a way to properly initialise the network so that it is able to immediately do the inference at around 20+ ms from the first time?

 

Thanks.

Ru Sern Yeoh
1 post / 0 new
For more complete information about compiler optimizations, see our Optimization Notice.