We are using Openvino in our Display solution. Currently we are facing some issues of which the details are mentioned below:
A single display is using multiple Inference models (Eg: 3) and we want to set all models with Hetero option (Eg: Hetero: GPU,CPU) with HW acceleration ON (HW acceleration being FFMPEG VAAPI or QSV).
In the above scenario we are seeing that output itself is not rendering at all and the Inference thread is in perpetual wait.When we did the same experiment with QSV builds, we are able to see the rendered output with inference outputs drawn.
We are using FFMPEG API to decode the given input instead of using OpenCV calls.
When an input is passed to ffmpeg (avformat_open_input) api, it will first create HW codec context using (av_hwdevice_ctx_create).
When we debugged the above scenario, we are seeing that Inference manager is stuck waiting in LoadNetwork ( InferenceEngine::InferencePlugin::LoadNetwork) in Inference thread.
Please note that av_hwdevice_ctx_create and LoadNetwork are working concurrently.
We are seeing that (av_hwdevice_ctx_create) thread is in waiting on lock for releasing resources. This call is common for both VAAPI and QSV.
av_hwdevice_ctx_create uses VAAPI driver calls in case of VAAPI build and Media SDK driver calls in case of QSV. We are seeing that this call is not getting locked in QSV for the above scenario and the rendering is happening smoothly.
Note: In the above problem scenario, if the first model is set to use simple CPU or GPU plugin and the remaining models to be set to Hetero, there are no problems and we are able to render all the outputs. This may be due to hardware context already created when the first model is given as a separate plugin instead of Hetero option.
1) Could you let us know how OpenVino is accessing CPU/GPU resources? is it similar to how VAAPI or QSV accessing the same?
2) Could you let us know whether there are any resource sharing happening between LoadNetwork API and in av_hwdevice_ctx_create in VAAPI which is causing a deadlock? (GDB back trace and the thread info is attached)
3) Is it valid scenario to access using Hetero for all models (instead of second model onwards) when decode is getting done using FFMPEG VAAPI ?
Attaching the gdb traces for the issue scenario