MKL 6.0 used under C# and .NET

MKL 6.0 used under C# and .NET

Our client (a bank) has used MKL 6.0 in a financial calculation library written in unmanaged C++. We have then linked this into a server application written in .Net (primarily C#). We accessed the library firstly through P/Invoke and more recently through IJW from managed C++.

Since many of the financial calculations are intensive, we make extensive use of the threading provided by .Net and, in general, spin up a new thread when a new user request comes in and terminate the thread when the calculation has completed (bear in mind that .Net threads aren't easily reuasble and do not necessarily have a one-to-one correspondance with OS threads). Now we have facilities to limit the maximum number of concurrent threads, but even so we are running into problems with MKL.

After someperiod of operation we get a message from the OMP Run-time library indicating that we have hit the maximum number of threads as specified in the KMP_ALL_THREADS env variable (32). We don't have this many threads concurrently executing, so I'm surprised to see this message.

All thoughts / ideas are welcome. I wonder if MKL requires threads to detach themselves and we are experiencing a cleanup problem.

Thanks in Advance
John Rayner

10 posts / 0 new
Last post
For more complete information about compiler optimizations, see our Optimization Notice.

I've not seen this before. What MKL functions are you using? How many processors are there on the system you are using (the sparse solver in MKL will use create as many threads as there are processors unless you explicitly set OMP_NUM_THREADS equal to 0).

-Todd

Sorry about the slow response Todd, but I've been trying (with limited success) to find out the answers to some of your questions. I'm afraid I don't know which MKL functions are being used - this code is in a proprietary library supplied to us and the dev team there have not responded to my enquiries. We are experiencing this problem primarily on our dev workstations which are dual processor machines. We haven't experienced this problem (yet) on our dev servers which are quad processor.

Out of interest, how does MKL maintain knowledge of the number of threads using it? Is there some explicit attach / detach process? Or do you use the OS-fired events in the DllMain proc?

- John

MKL uses OpenMP to manage threads. It won't take into account any threads which are created independent of OpenMP. In the absence of an OMP_NUM_THREADS environment variable, the maximum number of threads created by OpenMP is implementation dependent. For Intel OpenMP, it used to be the number of logical CPU's present. I understand there is also a KMP_SERIAL environment variable which can be set to stop this OpenMP from creating additional threads.

Tim,

Thanks for your response, but I'm still unclear about how I can attempt to resolve the issue I am facing. It is possibly down to the fact that I know practically nothing about MKL and OpenMP.

Perhaps we could try to come up with some scenarios under which it may be possible to cause the errorI sometimes get. For instance, if we're on a 2-way box the default values will be OMP_NUM_THREADS = 2 and KMP_ALL_THREADS = 32. Does this mean each MKL function which uses parallelism will use two threads? And there can only be 16 (i.e. 32 / 2) concurrent requests to this MKL function before we get the "maximum number of threads" error which we are seeing?

Thanks for your help and patience on this.

- John

John,

You have a good point, in that MKL is intended to be useful without attention to or knowledge of OpenMP details. We're already getting into issues where we might need expertise from the Threading forum. There is no default value of OMP_NUM_THREADS, but OpenMP should work as if this were set either to 1 or to the number of logical CPUs. If all the threading is done under OpenMP, any MKL function which sees the threading limit already taken up will not create any additional threads. Thus, I would suspect that your application is "hand threaded." It looks like OpenMP is getting fouled up, for example by an attempt to start multiple instances of OpenMP. I'm not certain whether hidden features like the KMP_SERIAL environment variable are intended to deal with this. If you can get the details together, it would be a good premier.intel.com issue, if only to bring forth expert advice. It's certainly common enough to use MKL in a hand threaded application which doesn't want MKL to create additional threads.

Tim,

Is there any information which I can provide which will help you (or anyone else reading this board) to suggest some kind of issue resolution?

- John

So we still have this issue and I still have very little clue as to its cause. Tim has mentioned possible duplicate initialisation of OpenMP. Does anybody have any idea about how this could happen? Or any other ideas? At all?

Hello

I was wondering if you managed to find a solution for your problem.

I am facing the same problem and found a workaround at my application's level.

You can contact me if you have any questions.

Take care

Message Edited by sinan.awad@ieee.org on 12-27-2005 12:24 PM

An application workaround was indeed what we ended up with as well.

What we did find out was that the DCOM objects which we were using were causing the garbage collector to run very slowly and often not keep up. Explicitly disposing as many objects as possible helped the garbage collector, but it also helped theKMP_ALL_THREADS issue considerably. So I think that the two issues were linked.

Cheers,
John

Leave a Comment

Please sign in to add a comment. Not a member? Join today