Analyzing Intel® MPI applications using Intel® Advisor

Many of today’s HPC applications use Intel® MPI to implement their parallelism. However, using Intel’s analyzer tools in a multi-process environment can be tricky. Intel® Advisor can be very helpful to maximize your vectorization, memory and threading performance. You can also use the Intel Advisor Roofline chart to visualize your performance bottlenecks. To analyze Intel MPI applications using Intel Advisor you should follow these steps to get the best value out of your results.

Remote analysis flow

First, collect data using the following command on the target:

mpirun -n 1 -gtool "advixe-cl -collect survey -no-auto-finalize -project-dir /user/test/vec_project:0" /user/test/vec_samples/vec_samples

Next, pack (optional) and retrieve results:

  1. Use the following command:
    advixe-cl --snapshot --project-dir /user/test/vec_project --pack --cache-sources --cache-binaries -- /tmp/my_proj_snapshot
  2. Copy vec_project or my_proj_snapshot to the host.

Finally, view on the host in the Intel Advisor GUI:

  1. Open the project using the following command: advixe-gui vec_project
  2. Open Project Properties to set up search paths.
  3. Open the result.

Collecting results using –no-auto-finalize

On some platforms like the Intel® Xeon Phi™ processor, result finalization may take long time. In such cases you can specify the –no-auto-finalize option so that finalization does not happen on your target. If you specify this option, then the results will finalize when you open them in the GUI.

MPI command line examples

Collect Survey data

mpirun -n 1 -gtool "advixe-cl -collect survey –no-auto-finalize -project-dir /user/test/vec_project:0" /user/test/vec_samples/vec_samples

To run non-Intel MPI, use the –trace-mpi option as follows:

mpirun -n 1 -gtool "advixe-cl -collect survey –trace-mpi –no-auto-finalize -project-dir /user/test/vec_project:0" /user/test/vec_samples/vec_samples

Collect Trip Counts data

mpirun -n 1 -gtool "advixe-cl -collect tripcounts –no-auto-finalize -project-dir /user/test/vec_project:0" /user/test/vec_samples/vec_samples

Collect Roofline data

Collecting the data for Roofline is a two step process. First run a survey collection and then run a trip counts collection where you specify that you also want flops data to be collected.

mpirun -n 1 -gtool "advixe-cl -collect survey –no-auto-finalize -project-dir /user/test/vec_project:0" /user/test/vec_samples/vec_samples
mpirun -n 1 -gtool "advixe-cl -collect tripcounts -flop –no-auto-finalize -project-dir /user/test/vec_project:0" /user/test/vec_samples/vec_samples

Collect Dependencies data

Note: You need to get the list of loops to analyze from either the report command or the Intel Advisor GUI.

mpirun -n 1 -gtool "advixe-cl -collect dependencies -mark-up-list=6,7,8,9,10 –no-auto-finalize -project-dir /user/test/vec_project:0" /user/test/vec_samples/vec_samples

Collect Memory Access Patterns data

Note: You need to get the list of loops to analyze from either the report command or the Intel Advisor GUI.

mpirun -n 1 -gtool "advixe-cl -collect map –no-auto-finalize -mark-up-list=6,7,8,9,10 -project-dir /user/test/vec_project:0" /user/test/vec_samples/vec_samples

Generating an MPI command line using the Intel Advisor GUI

For MPI applications, you need to collect your Intel Advisor results using the command line. We make this process easy by having the Intel Advisor GUI give you the precise command lines you need to run. There are two ways to get the command line; the first is using the Project Properties.

You have the option of choosing Intel MPI or another version of MPI. You can also specify the number of ranks you would like to run.

Generating the command line for Survey, Trip Counts analyses

You can also get the command line by clicking the Command Line button, right next to the Collect button as shown here. Once you have generated the command line, you would then need to cut and paste this line to a terminal window and run the command. It is sometimes helpful to specify the –no-auto-finalize option. If this option is specified, then the results will finalize when you open them in the GUI.

Here is a Survey command line:

Here is a Trip Counts command line:

Notice the –gtool option used above. This is an Intel MPI option; it allows our analyzers to only analyze the selected group of ranks. In this case we are only analyzing rank 0. If you were to not use –gtool and specified an MPI application with 10 ranks, then ten invocations of Intel Advisor would be launched.

Generating the command line for Memory Access Patterns analysis

To analyze the memory patterns in your application, you can select the loops in the Survey view.

Then click the Command Line button.

Generating the command line for Dependencies analysis

To check the dependencies of your loop, you again would need to select the loops you would like to analyze and then select the Command Line button.

Viewing the collected results in the GUI

Once you have collected your results, you will need to view them. The best way to do this is using the Intel Advisor GUI. If you specified the –no-auto-finalize option, it is important to open your project and then use Project Properties to set the paths to your binaries and sources. You need to do this before you open the results so we will be able to finalize them properly.

Then click the Survey tab.

Viewing the collected results without the GUI

You also have the option to view your Intel Advisor results without using the GUI. You can either generate a text report or a CSV report.

Text mode:

advixe-cl -report summary -project-dir ./advi -format text -report-output ./out/summary.txt
advixe-cl -report survey -project-dir ./advi -format text -report-output ./out/survey.txt
advixe-cl -report map -project-dir ./advi -format text -report-output ./out/map.txt
advixe-cl -report dependencies -project-dir ./advi -format text -report-output ./out/dependencies.txt

CSV mode:

advixe-cl -report summary -project-dir ./advi -format csv -csv-delimiter tab -report-output summary.csv
advixe-cl -report survey -project-dir ./advi -format csv -csv-delimiter tab -report-output survey.csv
advixe-cl -report map -project-dir ./advi -format csv -csv-delimiter tab -report-output map.csv
advixe-cl -report dependencies -project-dir ./advi -format csv -csv-delimiter tab -report-output dependencies.csv

Conclusion

Intel Advisor is a must-have tool for getting the most performance out of your MPI programs. To obtain Intel Advisor results for an MPI application:

  • Collect using CLI. You can generate a command line from the Intel Advisor GUI. If finalization is too slow, use the -no-auto-finalize option.
  • If you collect and view results on different machines, copy the result directory. You can pack the results into an archive to avoid additional configuration, if results were finalized.
  • Open the result in the Intel Advisor GUI. If the results were collected without finalization, configure search paths prior to opening the result.
For more complete information about compiler optimizations, see our Optimization Notice.