Thanks James. Well, I didn't mention this in my original post in order to avoid comingingly what appeared to be two separate problems, but I actually cannot build my test program using Intel MPI in conjunction with the GNU fortran compilers. When I try to build the program using mpif90 I get a long list of undefined reference errors, starting with this (and covering a broad spectrum of MPI calls that aren't used directly in the test program):

/home/pcarns/working/impi-4.0.3.008/intel64/lib/libmpigf.so: undefined reference to `PMPI_File_write_at_all'

/home/pcarns/working/impi-4.0.3.008/intel64/lib/libmpigf.so: undefined reference to `PMPI_File_read_at_all_end'

/home/pcarns/working/impi-4.0.3.008/intel64/lib/libmpigf.so: undefined reference to `PMPI_Unpack_external'

/home/pcarns/working/impi-4.0.3.008/intel64/lib/libmpigf.so: undefined reference to `PMPI_Win_get_errhandler'

/home/pcarns/working/impi-4.0.3.008/intel64/lib/libmpigf.so: undefined reference to `PMPI_Type_contiguous'

/home/pcarns/working/impi-4.0.3.008/intel64/lib/libmpigf.so: undefined reference to `PMPI_Type_create_subarray'

/home/pcarns/working/impi-4.0.3.008/intel64/lib/libmpigf.so: undefined reference to `PMPI_Win_start'

Thanks,

-Phil

Oh, and I should have mentioned that the Intel and GNU Fortran compilers are the only ones that I have available on the test workstation that I am using right now. I can't vouch for any of the other compilers. Thanks, -Phil

Hi James, To be precise, here is what I see when I try to use mpif90 with the GNU fortran compiler:

pcarns@carns-x220:~/working/darshan-examples$ source ~/working/impi-4.0.3.008/bin64/mpivars.sh

pcarns@carns-x220:~/working/darshan-examples$ which mpif90

/home/pcarns/working/impi-4.0.3.008/intel64/bin/mpif90

pcarns@carns-x220:~/working/darshan-examples$ mpif90 fperf.f -o fperf >& output.txt

pcarns@carns-x220:~/working/darshan-examples$ head -n 4 output.txt

/home/pcarns/working/impi-4.0.3.008/intel64/lib/libmpigf.so: undefined reference to `PMPI_File_write_at_all'

/home/pcarns/working/impi-4.0.3.008/intel64/lib/libmpigf.so: undefined reference to `PMPI_File_read_at_all_end'

/home/pcarns/working/impi-4.0.3.008/intel64/lib/libmpigf.so: undefined reference to `PMPI_Unpack_external'

/home/pcarns/working/impi-4.0.3.008/intel64/lib/libmpigf.so: undefined reference to `PMPI_Win_get_errhandler'

pcarns@carns-x220:~/working/darshan-examples$ mpif90 -show fperf.f -o fperf

gfortran -ldl -ldl -ldl -ldl fperf.f -o fperf -I/home/pcarns/working/impi-4.0.3.008/intel64/include/gfortran -I/home/pcarns/working/impi-4.0.3.008/intel64/include -L/home/pcarns/working/impi-4.0.3.008/intel64/lib -L/home/pcarns/working/impi-4.0.3.008/intel64/lib -Xlinker --enable-new-dtags -Xlinker -rpath -Xlinker /home/pcarns/working/impi-4.0.3.008/intel64/lib -Xlinker -rpath -Xlinker /opt/intel/mpi-rt/4.0.3 -lmpi -lmpigf -lmpigi -lpthread -lpthread -lpthread -lpthread -lrt

I was able to get the GNU version to build by using the following manual command line rather than mpif90:

gfortran fperf.f -o fperf -I/home/pcarns/working/impi-4.0.3.008/intel64/include/gfortran -I/home/pcarns/working/impi-4.0.3.008/intel64/include -L/home/pcarns/working/impi-4.0.3.008/intel64/lib -L/home/pcarns/working/impi-4.0.3.008/intel64/lib -ldl -lmpigi -lmpigf -lmpi -lpthread -lrt

Thank you very much, and I appreciate you trying it out first hand.
thanks, -Phil

Thanks James. I won't have time to look at this for a few days, but I will try it out later this week and report back. thanks, -Phil

Hi James, I haven't tried LD_PRELOAD wrapper yet, but I did confirm that I can set debugger breakpoints on functions following that naming convention (like mpi_file_open_). I have a few follow up questions, though, because this wasn't the profiling interface that I expected:

• Is there any shared entry point for C and Fortran bindings, or should a wrapper for the former catch MPI_File_open() while a wrapper for the latter catches mpi_file_open_()?
• Do these mpi_file_open_ etc. function names have exactly the same arguments as the C interface functions like MPI_File_open()?
• If I make a wrapper for mpi_file_open_, can it invoke an underlying pmpi_file_open_ after collecting profiling information? In other words, does it follow the convention of the MPI/PMPI function names in the C interface?
• Is the fortran function naming convention fairly stable across MPI releases (i.e., if one were to generate wrappers using those function names, will they likely still work in future Intel MPI releases?)?

Thanks!
-Phil

Ah, thanks for the clarification James. That makes sense. It didn't occur to me that I was just looking at the mangled version of the Fortran functions. I am definitely a novice at Fortran development! These Fortran functions/symbols are not what I was hoping for, though. In other major MPI implementations, there is at some level a set of C functions (usually it is just the normal mpi.h-defined C functions for the routines in question) that all bindings use under the covers. In other words, the underlying library is implemented as a collection of native C functions, and the various bindings are just invoking wrappers that account for argument differences and name mangling before calling those C functions. The side effect of this organization is that if you provide profiling wrappers for that C interface, you automatically catch the C++ and Fortran programs as well because they are funneling through that same underlying API. This may not be the case in Intel MPI, though. thanks, -Phil

I found some more information. I _can_ set a breakpoint on the PMPI_* version of the C functions rather than the MPI_* version of the C functions. For example:

Breakpoint 3, 0x00007ffff7916330 in PMPI_File_open ()

   from /home/pcarns/working/impi-4.0.3.008/intel64/lib/libmpi.so.4

(gdb) where

#0  0x00007ffff7916330 in PMPI_File_open ()

   from /home/pcarns/working/impi-4.0.3.008/intel64/lib/libmpi.so.4

#1  0x00007ffff7ec79b4 in pmpi_file_open__ ()

   from /home/pcarns/working/impi-4.0.3.008/intel64/lib/libmpigf.so.4

#2  0x000000000040383c in MAIN__ ()

Does this mean that the Fortran bindings are calling the PMPI functions directly instead of the intermediate MPI functions? I'm not sure why that would be the case, but it would explain the behavior that I am seeing.

thanks,
-Phil