Satisfactory support for multiple THREAD_FUNNELED ranks per node is a value added feature specific to each MPI. I think it is still under development for OpenMPI; with other open source MPIs you may have no way to do it except to make a script which gives each rank an appropriate set_affinity() or taskset command (or use 1 rank per node).
Your MPI is out of date; it may support affinity automatically only for Intel CPU models which were released prior to the MPI release date.
Many applications which use MKL are using mkl_sequential, which leaves the affinity entirely under the control of the application in the normal way. If you are using threaded MKL in THREAD_FUNNELED mode, the MKL should see the KMP_AFFINITY passed to the rank where it starts. With default setting of MKL_DYNAMIC, MKL will attempt to limit itself to 1 thread per core. MKL_NUM_THREADS and OMP_NESTED may also come into play. So I'm not certain you could say that KMP_AFFINITY has "direct effect." You could ask questions about MKL under Intel MPI on the HPC forum.

Tim,

I have seen several postings here relating to users wish a multi-threaded application that they also want to use MKL. The general problem the run into is dual thread pools.

One option for the user is to configure MKL to serial, this works well when the users multi-threaded code can make concurrent calls to MKL .AND. where the work done/sec exceeds that of running a serial app with MKL in parallel.

A second option is to run both in parallel and set the KMP_BLOCKTIME = 0 (or some small number)

The performance of MKL in parallel on large matrix sizes is superb. So if you do a substantial amount of work with large matrices then you would rather not run KMP serially... but what about the user application that can make use of some level of parallelization?

Note, I am not an MKL user/guru so I do not know all the tuning options.

It would seem appropriate to me to provide for an MKL_AFFINITY as well as the KMP_AFFINITY, and perhaps an initialization tuning function where the user can use general terms as to how to partition the cores between the application OpenMP threads and the MKL (OpenMP) threads. These could be either init-once or dynamic (TBD).

On a 4P system, the user might elect to ask for 1P for his application's OpenMP threads and 3Ps for MKL, (or other combinations such as:

n-threads for app, remainder for MKL
n-reserved application threads, m-total application threads, (MKL uses total - n)

Something like those options.

This functionality would provide for cooperative multi-threading.

Jim Dempsey

Michael,

I assume MKL will do its affinity pinning once, at the time of its thread pool initialization. Should this be the case, then you may have some success by:

program start
MKL call with exemplar array with KMP_AFFINITY set to your desired MKL configuration
then your first MPI call
remainder of program
end program

An exemplar array would be one that is large enough to require use of the number of (MKL) threads used within your application.

You may still have issues relating to multiple thread pools.

Jim Dempsey