Compiling multiple source code files with one source code contains OpenMP instructoins

Compiling multiple source code files with one source code contains OpenMP instructoins

I'm trying to compile multiple FORTRAN source files, where I applied some OpenMP directives to one of the source files. For instance :

The compilation flags :

COMPILE00='ifort -O3 -openmp -openmp_report -fpconstant -fp-model precise -fpe0 -traceback -ftrapuv'
COMPILE0='ifort -O3 -fpconstant -fp-model precise -fpe0 -traceback -ftrapuv'

The compiled files :

$COMPILE0 -c microprm.F90 modules.F90 
$COMPILE0 -c jernewf3p_A.F90 SMAX.F90 DE_MOTT.F90 twoinitm.F90 helek03.F90 
$COMPILE00 -c helek04.F90 
$COMPILE0 -c jernewf3p_melt.F90 dmin_G.F90 
$COMPILE0 -c submelt_condevap.F90

Linking :

$COMPILE00 -o TKE.x *.o -lm

So the source code helek04.F90 has some OpenMP directives. This code is being called from submelt_condevap.F90 which hasn't any OpenMP instructions. Is this correct compilation practice ? How should I compile it using the use omp_lib module ? Is it code-safe to compile only one file with OpenMP and link all the others with OpenMP ?

Compiling this way caused the compilation to aborted (some missed references). When compiling with $COMPILE00 for ALL the files, the simulation runs, but gets a floating point exception, which doesn't occur in the serial code.

Appreciate your comments,

Jack.

9 posts / 0 new
Last post
For more complete information about compiler optimizations, see our Optimization Notice.

In order to build procedures which will be called in parallel regions, in general they must either be declared RECURSIVE or built with -openmp or -auto set.  This has the effect of implementing local arrays without default SAVE effect, so as to avoid a cause of data race conditions.   Now that you have some buzz words, you could look up the prior posts on this subject.

Your floating point exception may be caused by a data race condition, where local data aren't updated correctly in each thread.   It's also possible that removing those SAVE defaults has exposed a bug in your code.   That would be a suspect if it doesn't run correctly when setting OMP_NUM_THREADS=1 or running with RECURSIVE or -auto without -openmp.

The only role of 'use omp_lib' is to supply explicit interface for OpenMP library calls.  This is independent of whether there are omp directives.  It's even possible to use omp_lib for the purpose of timing by omp_get_wtime() so as to make it work similar between linux and Windows, even when OpenMP is not in use (beyond the requirement to link the OpenMP library).  use omp_lib doesn't have side effects such as implying RECURSIVE or turning on checking for undefined data.

@Tim Prince - Thanks for your detailed replay.

Martyn Corden (Intel)'s picture

To the more general question:

It's simplest/safest to compile your whole application with -openmp, but not required. What is required is to build with -openmp any routine that might be called, directly or indirectly, from the routine containing OpenMP constructs, and also the main routine.. That ensures that local variables within such routines are threadsafe, provided you don't thwart that by the use of SAVE or -save. As Tim says, there are also other ways to do this, including compiling with -auto. After this, if you have large local arrays, you might find that you need to increase the main thread stack size (with ulimit -s) or the OpenMP (daughter) thread stack size (by setting or increasing OMP_STACKSIZE). Building the main program with -openmp ensures that the Fortran run-time library is initialized in threadsafe mode; again, there are other ways to achieve that, but this is simplest. The other requirement is to link the OpenMP run-time library. The easiest way to do this is to add -openmp to the link command line, if (as in your case) you are using the ifort driver for the link.

Martyn Corden - Thanks for adding to Tim's replay.

So as I understand, If I have a source file dependency as exampled below, I should compile at least the files along the dependency, i.e., MAIN.F90, ONE.F90, TWO.F90 (and if the MAIN would not had appeared in the dependency, it is recommend to link it (and compile?) it also using -openmp )  :

 <Program MAIN>

          <Call Subroutine ONE>

<End Program MAIN>

<Subroutine ONE>

         <Call Subroutine TWO> 

<End Subroutine ONE>

<Subroutine TWO>

  <!$OMP PARALLEL DO>

  <! DO SOMETHING>   

  <!$OMP END PARALLEL DO>  

<End Subroutine TWO>

 

jimdempseyatthecove's picture
REAL FUNCTION DISTANCE(A,B) RESULT(D)
  REAL :: A(3), B(3), TEMP(3)
  TEMP = A - B
  D = SQRT(DOTPRODUCT(C,C))
END FUNCTION DISTANCE

The problem is should the above be in a library (or lonely object file) compiled without -openmp, or -auto, or as recursive...

then the array TEMP is implicitly SAVE.

Should this function be called from within a parallel region, then all threads are sharing TEMP... meaning TEMP may get modified by other thread(s) prior to it being finished used by other threads.

By supply the aforementioned options, then TEMP is on stack, meaning each thread has a private copy.

EDIT 6/24:

REAL FUNCTION DISTANCE(A,B) RESULT(D)
  REAL :: A(3), B(3), TEMP(3) ! TEMP is local scope
  TEMP = A - B
  D = SQRT(DOTPRODUCT(TEMP,TEMP))
END FUNCTION DISTANCE

Jim Dempsey
 

www.quickthreadprogramming.com

@ Jim Dempsey - Thanks for clarifying that.

Martyn Corden (Intel)'s picture

In your example, MAIN needs to be compiled with -openmp so that the Fortran run-time library is initialized in threadsafe mode.

SUBROUTINE TWO  needs to be compiled with -openmp, and so does anything called from within the parallel region inside SUBROUTINE TWO, whether directly or indirectly, in order to ensure thread safety, as Jim and Tim have explained. SUBROUTINE ONE does not need to be compiled with -openmp, because it is never executed in a parallel context. Though as I said, it may be simpler to compile everything with -openmp.

You need to link with ifort -openmp in order to link the OpenMP run-time library, but also in order to link the threadsafe version of the Fortran run-time. This is distinct from compiling MAIN with -openmp to ensure threadsafe initialization. You need both to link to the threadsafe library and to initialize it in threadsafe mode - they are not the same thing.

You can find some of this, and other tips for using OpenMP from Fortran, at https://software.intel.com/en-us/articles/threading-fortran-applications-for-parallel-performance-on-multi-core-systems/

Martyn Corden  - Thanks for addressing my question.

Best regards,

Jack. 

Login to leave a comment.