Unable to generate trace file with mpirun

Unable to generate trace file with mpirun

Hello.  I've been attempting to follow the instructions on this page https://software.intel.com/sites/products/documentation/hpc/ics/itac/81/Getting_Started.htm using Intel mpirun but have either not been able to get the trace files to be generated or I cannot find them.  An example of my command line input is:

mpirun -trace -n 8 <app> <parameters>

Is there something I'm missing, or are the trace files generated somewhere other than your working directory by default?

Thanks for any input.

11 post / 0 nuovi
Ultimo contenuto
Per informazioni complete sulle ottimizzazioni del compilatore, consultare l'Avviso sull'ottimizzazione
Ritratto di James Tullos (Intel)

Do you have Intel® Trace Analyzer and Collector installed?  Have you sourced the itacvars.sh script?

Sincerely,
James Tullos
Technical Consulting Engineer
Intel® Cluster Tools

Thanks for the quick response.  I have that software installed and I have sourced the file /opt/intel/itac/8.1.4.045/intel64/bin/itacvars.csh (using csh shell).

Ritratto di James Tullos (Intel)

What is the output from the following:

env | grep I_MPI
env | grep VT
which mpirun

 

I_MPI_CC=icc
I_MPI_CXX=icc
I_MPI_F90=ifort

VT_MPI=impi4
VT_ROOT=/opt/intel//itac/8.1.4.045
VT_ARCH=intel64
VT_LIB_DIR=/opt/intel//itac/8.1.4.045/intel64/itac/lib_impi4
VT_SLIB_DIR=/opt/intel//itac/8.1.4.045/intel64/itac/slib_impi4
VT_MIC_SLIB_DIR=/opt/intel//itac/8.1.4.045/mic/itac/slib_impi4
VT_ADD_LIBS=-ldwarf -lelf -lvtunwind -lnsl -lm -ldl -lpthread

/nisprod/products1/intelmpi/em64t/4.0.2.003/bin64/mpirun

Ritratto di James Tullos (Intel)

Ok, I'll duplicate this and investigate this afternoon.  Is your application calling MPI_Init?  Also, try sourcing the /nisprod/products1/intelmpi/em64t/4.0.2.003/bin64/mpivars.sh script.

Yes, the application is calling MPI_Init.  I will try sourcing mpivars.sh and running again.  Are trace files generated as the application executes, or only after it finishes executing?  Thanks for your help so far.

Ritratto di James Tullos (Intel)

The trace files are generated when MPI_Finalize is called.  If your application is failing, we have a failsafe option you can use as well, which will save data in a temporary file and write the trace file when the application ends in any manner.

Still no luck after sourcing mpivars.csh.  One thing to note is that our application does not call MPI_Finalize directly but calls Petsc_Finalize which in turn should call MPI_Finalize.

Ritratto di James Tullos (Intel)
Best Reply

I'm able to get the trace file using those versions and environment variables (or at least the equivalent for my different paths).  Try running this command:

LD_PRELOAD=libVT.so mpirun ...

Is there a <program>.prot file in the current folder?  This should be generated when MPI_Init is called if you are linking to the Trace Collector library.

Setting LD_PRELOAD to libVT.so seemed to do the trick.  Now both a set of .stf files and a .prot file are being generated (the .prot file wasn't being created before either).  Thanks for you help!

Accedere per lasciare un commento.