Unable to generate trace file with mpirun

Unable to generate trace file with mpirun

Hello.  I've been attempting to follow the instructions on this page https://software.intel.com/sites/products/documentation/hpc/ics/itac/81/Getting_Started.htm using Intel mpirun but have either not been able to get the trace files to be generated or I cannot find them.  An example of my command line input is:

mpirun -trace -n 8 <app> <parameters>

Is there something I'm missing, or are the trace files generated somewhere other than your working directory by default?

Thanks for any input.

11 帖子 / 0 全新
最新文章
如需更全面地了解编译器优化,请参阅优化注意事项
James Tullos (Intel)的头像

Do you have Intel® Trace Analyzer and Collector installed?  Have you sourced the itacvars.sh script?

Sincerely,
James Tullos
Technical Consulting Engineer
Intel® Cluster Tools

Thanks for the quick response.  I have that software installed and I have sourced the file /opt/intel/itac/8.1.4.045/intel64/bin/itacvars.csh (using csh shell).

James Tullos (Intel)的头像

What is the output from the following:

env | grep I_MPI
env | grep VT
which mpirun

 

I_MPI_CC=icc
I_MPI_CXX=icc
I_MPI_F90=ifort

VT_MPI=impi4
VT_ROOT=/opt/intel//itac/8.1.4.045
VT_ARCH=intel64
VT_LIB_DIR=/opt/intel//itac/8.1.4.045/intel64/itac/lib_impi4
VT_SLIB_DIR=/opt/intel//itac/8.1.4.045/intel64/itac/slib_impi4
VT_MIC_SLIB_DIR=/opt/intel//itac/8.1.4.045/mic/itac/slib_impi4
VT_ADD_LIBS=-ldwarf -lelf -lvtunwind -lnsl -lm -ldl -lpthread

/nisprod/products1/intelmpi/em64t/4.0.2.003/bin64/mpirun

James Tullos (Intel)的头像

Ok, I'll duplicate this and investigate this afternoon.  Is your application calling MPI_Init?  Also, try sourcing the /nisprod/products1/intelmpi/em64t/4.0.2.003/bin64/mpivars.sh script.

Yes, the application is calling MPI_Init.  I will try sourcing mpivars.sh and running again.  Are trace files generated as the application executes, or only after it finishes executing?  Thanks for your help so far.

James Tullos (Intel)的头像

The trace files are generated when MPI_Finalize is called.  If your application is failing, we have a failsafe option you can use as well, which will save data in a temporary file and write the trace file when the application ends in any manner.

Still no luck after sourcing mpivars.csh.  One thing to note is that our application does not call MPI_Finalize directly but calls Petsc_Finalize which in turn should call MPI_Finalize.

James Tullos (Intel)的头像
Best Reply

I'm able to get the trace file using those versions and environment variables (or at least the equivalent for my different paths).  Try running this command:

LD_PRELOAD=libVT.so mpirun ...

Is there a <program>.prot file in the current folder?  This should be generated when MPI_Init is called if you are linking to the Trace Collector library.

Setting LD_PRELOAD to libVT.so seemed to do the trick.  Now both a set of .stf files and a .prot file are being generated (the .prot file wasn't being created before either).  Thanks for you help!

登陆并发表评论。