[Bug report] Memory leak in CPU plugin

[Bug report] Memory leak in CPU plugin

Hello,

I am experiencing a big memory leak with the attached model. It appears that ExecutableNetwork does not deallocate everything when it is destructed.

I have only tried the cpu plugin, so I do not know if it is present in other plugins as well.

I am running the latest version of openvino (2018.5.456) on a Windows 10 machine with a Xeon W2133 cpu.

Find attached the problematic model and a code snippet to reproduce.

When executing the code, the commit size observed in Windows Task Manager continues to increase by approximately 50 MB for every iteration.

 

Regards,

Thomas

AttachmentSize
Downloadapplication/zip openvinoMemoryBug.zip2.62 MB
39 posts / 0 new
Last post
For more complete information about compiler optimizations, see our Optimization Notice.

Bump

Hello Thomas. Here is the code in your main.cpp:

while (true)
{
    std::cout << "Press key to continue." << std::endl;
    std::cin.ignore();

    std::cout << "Loading inference plugin" << std::endl;
    auto plugin = PluginDispatcher({ L"" }).getPluginByDevice("CPU");

    CNNNetReader networkReader;
    networkReader.ReadNetwork("saved.xml");
    networkReader.ReadWeights("saved.bin");

    auto cnnNetwork = networkReader.getNetwork();
    cnnNetwork.setBatchSize(1);

    auto executableNetwork = plugin.LoadNetwork(cnnNetwork, {});
}

Your code has never left scope and all those objects are created via smart pointers. Why would you expect memory to be released while stuck in this while loop ? 

Hi,

The smart-pointers are local variables inside the loop and thus go out of scope at the end of each iteration of the loop. See for example here: https://stackoverflow.com/questions/6403055/object-destruction-in-c

FWIW we are also seeing a statics order of destruction issue; not sure if this is related. Let's further debug this.

Hello Thomas, (1) does not compile - came from your sample. However (2) does compile - comes from the OpenVino samples.

1) auto plugin = PluginDispatcher({ L"" }).getPluginByDevice("CPU")

2) InferencePlugin plugin = PluginDispatcher({ FLAGS_pp, "../../../lib/intel64" , "" }).getPluginByDevice(FLAGS_d);

Can you kindly clarify ?

Thanks,

Shubha

This is related to whether the UNICODE preprocessor symbol is defined. The classification sample compiles without UNICODE and in this case the line should be changed to read

auto plugin = PluginDispatcher({ "" }).getPluginByDevice("CPU");

Or just use the one from the classification sample. Both will provoke the memory leak.

PS: I used to compile with UNICODE set, but found a scenario where this gave rise to an access violation (segfault) when calling from some c# code. There is a post elsewhere on this forum mentioning that. I now compile without UNICODE and still experience the memory leak.

Were you able to reproduce this?

Hi again,

I have investigated this and have found the location of the leak. It is a thread-local work buffer in MKL-DNN that doesn't get deallocated because OpenVino calls the constructors and destructors of MKL-DNN primitives from different threads.

Specifically, the global_scratchpad_t object in https://github.com/opencv/dldt/blob/2018/inference-engine/thirdparty/mkl... is allocated by a task created here https://github.com/opencv/dldt/blob/2018/inference-engine/src/mkldnn_plu... (and this allocation may therefore happen off the main thread). The scratchpad is deallocated when the InferenceEngine::ExecutableNetwork object in main() is destructed, which happens on the main thread. However, MKL-DNN uses thread local buffers and assumes that those two calls happen from the same thread.

The memory leak can be fixed by compiling MKL-DNN with preprocessor definition MKLDNN_ENABLE_CONCURRENT_EXEC. That may, however, increase memory usage (see https://github.com/intel/mkl-dnn/blob/master/cmake/options.cmake#L33). The real solution is probably to make sure that MKL-DNN primitives are constructed and destructed on the same thread.

- Thomas

Good find Thomas, thank you! Hopefully this gets fixed soon.

I am going to try with MKLDNN_ENABLE_CONCURRENT_EXEC and keep an eye on memory usage to see if our issues get resolved.

cheers

nikos

Hi Thomas, with your help I have reproduced this issue and filed a bug. Thank You !

Shubha

Hi Shubha,

Do you know if this is fixed in the 2019 R1 release?

Dear Thomas, yes this leak has been fixed. Please download 2019 R1 and give it a test drive !

Thanks for using OpenVino !

Shubha

Hi Shubha,

I have just tried 2019 R1 and unfortunately I am still seeing this memory leak. Can you update the bug report with this information?

Thanks,

Thomas

Dear Thomas, 

Wow. Sorry for this. I will certainly update the bug report accordingly.

Thanks,

Shubha

As far as I know the google gflags library has memory leak problem

Dear Thomas, this is not an OpenVino bug. Please review the following code:

//There is some leak per each LoadLibrary <-> FreeLibrary call on Windows. 

int main(int argc, char *argv[]) {

    LPCTSTR pluginName = "Full path to MKLDNNPlugin.dll";

    int c = 100;
    do { 

{         HMODULE object = LoadLibrary(pluginName); 

        FreeLibrary(object); 

         c–; 

    } 

 while (c > 0);
}

To avoid it, just to load plugin once:

InferencePlugin plugin = PluginDispatcher().getPluginByDevice("CPU");

 

Hi Shubha, Could you answer my question here?

Quote:

Shubha R. (Intel) wrote:

Dear Thomas, 

Wow. Sorry for this. I will certainly update the bug report accordingly.

Thanks,

Shubha

Hi Shubha,

Loading the plugin just once doesn't fix the leak either.

Any update on this?

Is it an OpenVino bug or not?

If yes, ETA on a fix?

thanks!

Nikos

Hi.

Any updates on this?

Thanks, Thomas

Dear Lundgaard, Thomas,

There is no such leak in OpenVino. It could be coming from the gflags library or somewhere else. The leak is not within Inference Engine.

Thanks,

Shubha

In a previous post I pointed out the location of the memory leak. Looking at the source code for OpenVino 2019 R1.1 I can see that it is still not fixed.

I can reproduce this issue without gflags, so gflags is NOT the source of the leak.

I have also tried just loading the CPU plugin once, and still observe the leak. So this is NOT caused by Windows and multiple calls to LoadLibrary and FreeLibrary.

 

You previously acknowledged this as a bug in OpenVino. It is still not fixed. Can you please forward this information to the developers?

 

Thank you, Thomas

Dear Lundgaard, Thomas,

I have forwarded this to OpenVino Developers. They state that there is no memory leak. I can certainly try again.

Thanks,

Shubha

 

Hi Shubha,

I would appreciate it if you send it to the developers again. If they persist that this is not a leak in openvino, it would be great if they can comment on my findings described in one of the posts above (https://software.intel.com/en-us/forums/computer-vision/topic/804912#com...).

Thank you,

Thomas

Dearest Lundgaard, Thomas,

OpenVino 2019R2 should be released soon. I don't know exactly when, however. Here's my advice to you - and it kinda sucks because it increases your workload but it will be a surefire way to convince the developers.

Using https://github.com/opencv/dldt please build a self contained little project which demonstrates the issue without gflags. And I know that you've done this before (a long time ago) - I used your little project in fact to file a bug. Don't do this just yet however. Please wait for the R2 release.

Please see my detailed response to this github forum post for hints on how to build a self-contained project.

Would that be OK for you ? 

Thanks,

Shubha

 

Hi,

Yes I can do that when 2019R2 is released.

I wont build OpenVino from scratch as there is no need for that (and my trial of the intel c++ compiler has expired). I will just compile a small example application against the OpenVino binaries you distribute -- just as I did last time.

Thanks, Thomas

Dearest Lundgaard, Thomas,

That would be great.

Thanks !

Shubha

Hi Shubha,

I have tried with 2019 R2 and still get the memory leak. The attached project reproduces it.

For every iteration in the loop I get about 8 MB of increase in memory usage.

I hope this will be fixed in 2019 R3.

 

Best regards,

Thomas

Attachments: 

AttachmentSize
Downloadapplication/zip openvinoleak.zip2.62 MB

Dear Lundgaard, Thomas,

Thanks for the attached project. I will see if I can reproduce the memory leak on R2.

Shubha

 

Dear Lundgaard, Thomas,

Please see the attached image. I have run your sample program for over 4 minutes on OpenVino 2019R2.01 and I could not produce a memory leak. What I see is a constant memory usage of around 12 MB. If there were a memory leak, the memory usage would have increased during a 4 minute period but it didn't - it simply stayed constant. So there is no memory leak.

Shubha

Attachments: 

AttachmentSize
Downloadimage/png no_leak_2.png140.88 KB

Hi,

It appears that you never got it to run inference. You have to repeatedly press a key in the console window of the application. I made it like that so the program wont eat up all your memory. Alternatively you can remove the line "std::cin.ignore();" in main.cpp.

As you can see on the attached screenshot, there is indeed a memory leak on my machine.

- Thomas

Attachments: 

Dear Lundgaard, Thomas,

OK I didn't repeatedly hit a key in the console window. Let me try again. Looking at your image, I do see a tiny (not big) leak.

Thanks,

Shubha

 

Hi there,

just to add to Thomas findings, I'm experiencig a very similar leak under Linux.

Recently I switched to 2019R2.0.1 but still identical behavior.

Could you please rise the priority of investigation on this issue?

 

Thanks a lot in advance!

E.

Dear Marchetto, Enrico,

I would be glad to prioritize this but what are you specifically doing to cause your memory leak ? Thomas is literally doing just the short piece of code which follows. So is your situation different ? And if it is, can you kindly attach code to this ticket which demonstrates the problem ?

int main(char argv[])
{
	std::cout << "Loading inference plugin" << std::endl;
	auto plugin = PluginDispatcher({ "" }).getPluginByDevice("CPU");

	CNNNetReader networkReader;
	networkReader.ReadNetwork("saved.xml");
	networkReader.ReadWeights("saved.bin");

	auto cnnNetwork = networkReader.getNetwork();
	cnnNetwork.setBatchSize(1);

	while (true)
	{
		std::cout << "Press key to continue." << std::endl;
		std::cin.ignore();

		auto executableNetwork = plugin.LoadNetwork(cnnNetwork, {});
	}
}

 Thanks,

Shubha

Shubha,

Were you able to reproduce this?

Dear Lundgaard, Thomas,

OpenVino 2019R3 should arrive in about a month (or less). I tried your code (hitting enter like you said for about 3 minutes) and I could not produce the leak on R3.  At this point, it makes little sense to go backwards and try and repro it on R2. I couldn't rig Visual Studio 2017's Memory Analyzer since this required my hitting enter and within Visual Studio the code just exited (failed). I am not sure how I can hit enter through Visual Studio - i tried researching this to no avail. I tried attaching to a process and launching an exe (I tried different ways) but Memory Profiler within Visual Studio is only available by launching the app. Remember last time I did this, I did it incorrectly as i didn't hit enter repeatedly as you later instructed.

But what I did finally was run your app at the command-line hitting enter repeatedly for about 3 minutes, and I used Resource Monitor. And I saw absolutely no evidence of a memory leak on R3. So it looks like R3 fixed it.

If you still see it in R3, please let us know.

Hope it helps,

Thanks

Shubha

 

Quote:

Shubha R. (Intel) wrote:

Dear Marchetto, Enrico,

I would be glad to prioritize this but what are you specifically doing to cause your memory leak ? Thomas is literally doing just the short piece of code which follows. So is your situation different ? And if it is, can you kindly attach code to this ticket which demonstrates the problem ?

int main(char argv[])
{
	std::cout << "Loading inference plugin" << std::endl;
	auto plugin = PluginDispatcher({ "" }).getPluginByDevice("CPU");

	CNNNetReader networkReader;
	networkReader.ReadNetwork("saved.xml");
	networkReader.ReadWeights("saved.bin");

	auto cnnNetwork = networkReader.getNetwork();
	cnnNetwork.setBatchSize(1);

	while (true)
	{
		std::cout << "Press key to continue." << std::endl;
		std::cin.ignore();

		auto executableNetwork = plugin.LoadNetwork(cnnNetwork, {});
	}
}

 Thanks,

Shubha

 

Dear Shubha,

I'm running a minimal code sample actually very similar to Thomas's one.

I see that you are not able to reproduce the leak with R3, which could be good news. Could you have a feedback from the developers to know if they actually addressed the issue?

Thanks a lot!

Best,

Enrico

 

Dearest Enrico,

The developers rely on me to demonstrate that there is indeed a memory leak (using code such as Thomas's above) and so far, I've not been able to demonstrate this on R3.

Hope it helps,

Shubha

Leave a Comment

Please sign in to add a comment. Not a member? Join today