User Guide

  • 12/15/2019
  • Public Content

Collective Operations Time per Rank

Key:
-c [--collop_time_per_rank]
Available options:
-C [--communicators]
,
-M [--comm_id]
,
-E [--internal_communicators]
This charts shows the time each rank spent in MPI collective operations. Extend the chart using available options:

Example 1 (-c)

$ aps-report ./aps_result_<postfix> -c
| Collective Operations Time and Communicators for all ranks |-------------------------------------------------------------- | Rank     LifeTime(sec) CollOp Time(sec)  CollOp Time(%) |---------------------------------------------------------   0003              3.48             2.32           66.73   0002              3.48             2.21           63.38   0001              3.48             2.08           59.72   0000              4.53             2.52           55.67 |========================================================= | TOTAL            14.96             9.12           60.98 | AVG               3.74             2.28           60.98 |

Example 2 (-c -C -M)

This charts shows the time each rank spent in MPI collective operation in a specific communicator.
$ aps-report ./aps_result_<postfix> -c -C -M 1
| Collective Operations Time and Communicators for all ranks |-------------------------------------------------------------- |-------------------------------------------------------------------------------------------------------------------------- | Function Time(sec) Time(%) Volume(MB) Volume(%) Calls AvrMsgSize(B) |-------------------------------------------------------------------------------------------------------------------------- MPI_Allreduce 1.95 34.12 11087.89 100.00 130710 88949 MPI_Barrier 0.45 7.81 0.00 0.00 134990 0 | [filtered out 2 lines] |========================================================================================================================== |-------------------------------------------------------------------------------------------------------------------------- | Function Time(sec) Time(%) Volume(MB) Volume(%) Calls MsgSize(B) |-------------------------------------------------------------------------------------------------------------------------- MPI_Allreduce 1.95 34.12 11087.89 100.00 130710 ALL Comm.: Id Size 1 4 1.95 100.00 11087.89 100.00 130710 ALL |-------------------------------------------------------------------------------------------------------------------------- MPI_Barrier 0.45 7.81 0.00 100.00 134990 ALL Comm.: Id Size 1 4 0.45 100.00 0.00 100.00 134990 ALL |-------------------------------------------------------------------------------------------------------------------------- MPI_Gather 0.00 0.01 0.00 100.00 184 ALL Comm.: Id Size 1 4 0.00 100.00 0.00 100.00 184 ALL |-------------------------------------------------------------------------------------------------------------------------- MPI_Bcast 0.00 0.01 0.00 100.00 90 ALL Comm.: Id Size 1 4 0.00 100.00 0.00 100.00 90 ALL |==========================================================================================================================

Example 3 (-c -C)

This chart shows the time spent by each MPI collective operation in all communicators.
$ aps-report ./aps_result_<postfix> -c -C -E
Collective Operations Time and Communicators for all ranks |-------------------------------------------------------------- |-------------------------------------------------------------------------------------------------------------------------- | Function Time(sec) Time(%) Volume(MB) Volume(%) Calls AvrMsgSize(B) |-------------------------------------------------------------------------------------------------------------------------- MPI_Allreduce 3.44 23.26 13859.89 71.43 197658 73527 MPI_Barrier 2.64 17.87 0.00 0.00 626018 0 MPI_Scatter 2.31 15.61 2771.99 14.29 219396 13248 MPI_Reduce 1.64 11.09 2771.97 14.29 195384 14876 | [filtered out 2 lines] |========================================================================================================================== |-------------------------------------------------------------------------------------------------------------------------- | Function Time(sec) Time(%) Volume(MB) Volume(%) Calls MsgSize(B) |-------------------------------------------------------------------------------------------------------------------------- MPI_Allreduce 3.44 23.26 13859.89 100.00 197658 ALL Comm.: Id Size 4611686018431582656 4 1.62 47.12 11087.88 80.00 130520 ALL 4611686018431582208 4 1.36 39.53 0.01 0.00 744 ALL 4611686018429485520 2 0.46 13.23 2771.97 20.00 65260 ALL | [filtered out 16 lines] |-------------------------------------------------------------------------------------------------------------------------- MPI_Barrier 2.64 17.87 0.00 100.00 626018 ALL Comm.: Id Size 4611686018431582208 4 1.08 40.98 0.00 100.00 744 ALL 4611686018429485456 2 0.59 22.47 0.00 100.00 146848 ALL 4611686018431582608 4 0.27 10.34 0.00 100.00 73424 ALL 4611686018429485552 2 0.25 9.38 0.00 100.00 130792 ALL 4611686018431582704 4 0.21 8.02 0.00 100.00 65396 ALL 4611686018431582656 4 0.18 6.69 0.00 100.00 130792 ALL 4611686018429485520 2 0.05 1.73 0.00 100.00 65396 ALL | [filtered out 8 lines] |--------------------------------------------------------------------------------------------------------------------------

Chart Entries

-c and -c -M Columns
-c -C Columns
Rows
Rank
Rank number
LifeTime(sec)
Total execution time of the rank (in seconds)
CollOpTime(sec)
Time spent in MPI collective operations (in seconds)
CollOpTime(%)
Percentage of the MPI collective operations time in the application lifetime
Rank
Rank number
LifeTime(sec)
Total execution time of the rank (in seconds)
CollOpTime(sec)
Time spent in MPI collective operations (in seconds)
CollOpTime(%)
Percentage of the MPI collective operations time in the application lifetime
Function
Function name
Time(sec)
Total time of the function execution in all ranks (in seconds)
Time(%)
Percentage of the function execution time in the application lifetime
Idle(sec)
The time when the collective operation was idle (in seconds)
Idle(%)
Percentage of the time when the operation was idle
Volume(MB)
Amount of data transferred by the function in all ranks (in megabytes)
Volume(%)
Percentage of the data transferred by the function
Calls
Number of calls of this function in all ranks
AvrMsgSize(B)
Average size of messages
TOTAL
Total values for all columns
AVG
Average values for all columns
 

Product and Performance Information

1

Intel's compilers may or may not optimize to the same degree for non-Intel microprocessors for optimizations that are not unique to Intel microprocessors. These optimizations include SSE2, SSE3, and SSSE3 instruction sets and other optimizations. Intel does not guarantee the availability, functionality, or effectiveness of any optimization on microprocessors not manufactured by Intel. Microprocessor-dependent optimizations in this product are intended for use with Intel microprocessors. Certain optimizations not specific to Intel microarchitecture are reserved for Intel microprocessors. Please refer to the applicable product User and Reference Guides for more information regarding the specific instruction sets covered by this notice.

Notice revision #20110804