There's nothing visibly wrong with your first version of OpenMP parallel do. An Intel OpenMP thread team stays active for the time interval set by KMP_BLOCKTIME (default 0.200 seconds), so it's difficult to understand what you expected to accomplish differently with the second version. Your openmp-profile report would shed more light on what is happening in libiomp5. You could simply set LD_PRELOAD=/libiompprof5.so Did you consider trying Inspector to see if you have unintended data sharing between threads?
Do you get any parallel speedup from the OpenMP internal to MKL DGETRS when you turn off OpenMP in your calling code?

Petros,

Some portions of code work best when parallization occurs inside MKL while others work best when parallization is performed outside MKL (with MKL restricted to one thread). Performing parallization both inside and outside MKL usually does not work well. From MKL docs:

WARNING.

It is not recommended to simultaneously parallelize your program and employ the

Intel MKL internal threading because this will slow down the performance. Note that

in case "d" above, DFT computation is automatically initiated in a single-threading

mode.

Try the following experiment

!$omp parallel
call mkl_set_num_threads(1) ! see note below code snip
do while(not converged)

!$omp single
CODE THAT HAS TO BE EXECUTED IN SERIAL (prepare data)
!$omp end single

!$omp do
do i=1, "several thousands"
CALL DGETRS(data of i)
enddo
!$omp end do

!$omp single
CODE THAT HAS TO BE EXECUTED IN SERIAL (check convergence)
!$omp end single

enddo
!$omp end parallel

Note, TimP may be able to answer this, what I do not know is if mkl_set_num_threads(n) is global or thread local. Due to this uncertanty, this is the reason for placing " call mkl_set_num_threads(1)" inside the parallel region (it should be benign to reset it repeatedly).

Also note, if you have other sections of code the work better with MKL internally parallized you will have to reset the MKL number of threads. And there are other, more complex, ways of managing threads between your application and MKL.

As to if DGETRS works best with parallization inside MKL or outside MKL I cannot say. The experiment should be easy to run.

From looking at the chart a significant portion of time is spent inside MKL in start_thread. This would seem to indicate that for your usage of DGETRS that parallization outside of MKL might be favorable.

Jim Dempsey

I suppose the reason for supporting MKL_NUM_THREADS as well as OMP_NUM_THREADS is to permit independent specification for MKL functions in an application which also has its own OpenMP parallel. I haven't seen documentation, but I suppose either set_num_threads option takes effect for the next parallel region (next MKL function call counts); I doubt that mkl_set_num_threads is thread local, but it's an interesting question.
As Jim points out, calling threaded MKL inside a parallel region could be problematical, and the MKL team doesn't recommend ways of doing it, not that you couldn't experiment with this form of OMP_NESTED. Among other things, you would want to set omp parallel num_threads and MKL_NUM_THREADS such that the product doesn't oversubscribe the resources. Affinity could be tricky, even with combined use of MKL_AFFINITY and KMP_AFFINITY.
It may be that extreme MKL thread overhead is due to an excessive total number of threads. By setting MKL_NUM_THREADS=1, you should get nearly as good performance as with mkl_sequential library linkage. As Jim said, it's common for parallelism outside MKL to prove superior to inside MKL, but this depends on your data set, how it fits cache, and I suppose for DGETRS whether the number of right hand sides is sufficient to use all your cores efficiently.

The decision as to use parallelization inside MKL or outside MKL should be taken on a case by case basis as opposed to functionality by functionality basis. Take for example matrix multiply. If you were to have many different matricies to multiply I would postulate that benieth some size parallization would be more suitable outside MKL and larger than some size parallelization inside MKL would be better. Also, on some configurations, say multi-processor/socket a combination were each socket runs a seperate thread that calls MKL which then parallelizes within the socket although I do not know if MKL has that degree of flexibility (it may have).

Jim Dempsey

If different threads are updating in the same pair of cache lines, it can create a large false sharing overhead. It may be improved if you can arrange (e.g. by setting affinity) that most cache line sharing is local to a CPU.
This is a case where hyperthreading may tolerate 2 threads on the same core updating elements less than 64 bytes apart.