Unable to generate trace file with mpirun

Unable to generate trace file with mpirun

Hello.  I've been attempting to follow the instructions on this page https://software.intel.com/sites/products/documentation/hpc/ics/itac/81/Getting_Started.htm using Intel mpirun but have either not been able to get the trace files to be generated or I cannot find them.  An example of my command line input is:

mpirun -trace -n 8 <app> <parameters>

Is there something I'm missing, or are the trace files generated somewhere other than your working directory by default?

Thanks for any input.

11 posts / 0 new
Last post
For more complete information about compiler optimizations, see our Optimization Notice.

Do you have Intel® Trace Analyzer and Collector installed?  Have you sourced the itacvars.sh script?

James Tullos
Technical Consulting Engineer
Intel® Cluster Tools

Thanks for the quick response.  I have that software installed and I have sourced the file /opt/intel/itac/ (using csh shell).

What is the output from the following:

env | grep I_MPI
env | grep VT
which mpirun



VT_ADD_LIBS=-ldwarf -lelf -lvtunwind -lnsl -lm -ldl -lpthread


Ok, I'll duplicate this and investigate this afternoon.  Is your application calling MPI_Init?  Also, try sourcing the /nisprod/products1/intelmpi/em64t/ script.

Yes, the application is calling MPI_Init.  I will try sourcing mpivars.sh and running again.  Are trace files generated as the application executes, or only after it finishes executing?  Thanks for your help so far.

The trace files are generated when MPI_Finalize is called.  If your application is failing, we have a failsafe option you can use as well, which will save data in a temporary file and write the trace file when the application ends in any manner.

Still no luck after sourcing mpivars.csh.  One thing to note is that our application does not call MPI_Finalize directly but calls Petsc_Finalize which in turn should call MPI_Finalize.

Best Reply

I'm able to get the trace file using those versions and environment variables (or at least the equivalent for my different paths).  Try running this command:

LD_PRELOAD=libVT.so mpirun ...

Is there a <program>.prot file in the current folder?  This should be generated when MPI_Init is called if you are linking to the Trace Collector library.

Setting LD_PRELOAD to libVT.so seemed to do the trick.  Now both a set of .stf files and a .prot file are being generated (the .prot file wasn't being created before either).  Thanks for you help!

Leave a Comment

Please sign in to add a comment. Not a member? Join today