Recording Communication Events

These are API calls that allow logging of message send and receive and MPI-style collective operations. Because they are modeled after MPI operations, they use the same kind of communicator to define the context for the operation.

enum _VT_CommIDs

Logging send/receive events evaluates the rank local within the active communicator, and matches events only if they take place in the same communicator (in other words, it is the same behavior as in MPI).

Defining new communicators is not supported, but the predefined ones can be used.

Enumerator Description
VT_COMM_INVALID

Invalid ID, do not pass to Intel® Trace Collector

VT_COMM_WORLD

Global ranks are the same as local ones

VT_COMM_SELF

Communicator that only contains the active process

 

VT_log_sendmsg

int VT_log_sendmsg(int other_rank, int count, int tag, int commid, int sclhandle)

Description

Logs sending of a message.

Fortran

VTLOGSENDMSG(other_rank, count, tag, commid, sclhandle, ierr)

Parameters

my_rank rank of the sending process

other_rank rank of the target process

count number of bytes sent

tag tag of the message

commid numeric ID for the communicator (VT_COMM_WORLD, VT_COMM_SELF, or see VT_commdef())

sclhandle handle as defined by VT_scldef, or VT_NOSCL

Return values

Returns error code.

 

VT_log_recvmsg

int VT_log_recvmsg(int other_rank, int count, int tag, int commid, int sclhandle)

Description

Logs receiving of a message.

Fortran

VTLOGRECVMSG(other_rank, count, tag, commid, sclhandle, ierr)

Parameters

my_rank rank of the receiving process

other_rank rank of the source process

count number of bytes sent

tag tag of the message

commid numeric ID for the communicator (VT_COMM_WORLD, VT_COMM_SELF, or see VT_commdef())

sclhandle handle as defined by VT_scldef, or VT_NOSCL

Return values

Returns error code.

 

The next three calls require a little extra care, because they generate events that not only have a time stamp, but also a duration. This means that you need to take a time stamp first, then do the operation and finally log the event.

VT_log_msgevent

int VT_log_msgevent(int sender, int receiver, int count, int tag, int commid, double sendts, int sendscl, int recvscl)

Description

Logs sending and receiving of a message.

Fortran

VTLOGMSGEVENT(sender, receiver, count, tag, commid, sendts, sendscl, recvscl, ierr)

Parameters

sender rank of the sending process

receiver rank of the target process

count number of bytes sent

tag tag of the message

commid numeric ID for the communicator (VT_COMM_WORLD, VT_COMM_SELF, or see VT_commdef())

sendts time stamp obtained with VT_timestamp()

sendscl handle as defined by VT_scldef() for the source code location where the message was sent, or VT_NOSCL

recvscl the same for the receive location

Return values

Returns error code.

 

VT_log_op

int VT_log_op(int opid, int commid, int root, int bsend, int brecv, double startts, int sclhandle)

Description

Logs the duration and amount of transferred data of an operation for one process.

Fortran

VTLOGOP(opid, commid, root, bsend, brecv, startts, sclhandle, ierr)

Parameters

opid id of the operation; must be one of the predefined constants in enum _VT_OpTypes

commid numeric ID for the communicator; see VT_log_sendmsg() for valid numbers

root rank of the root process in the communicator (ignored for operations without root, must still be valid, though)

bsend bytes sent by process (ignored for operations that send no data)

brecv bytes received by process (ignored for operations that receive no data)

startts the start time of the operation (as returned by VT_timestamp())

sclhandle handle as defined by VT_scldef, or VT_NOSCL

Return values

Returns error code.

 

VT_log_opevent

int VT_log_opevent(int opid, int commid, int root, int numprocs, int _ bsend, int _ brecv, double _ startts, int sclhandle)

Description

Logs the duration and amount of transferred data of an operation for all involved processes at once.

Intel® Trace Collector knows which processes send and receive data in each operation. Unused byte counts are ignored when writing the trace, so they can be left uninitialized, but NULL is not allowed as array address even if no entry is used at all.

Fortran

VTLOGOPEVENT(opid, commid, root, numprocs, bsend, brecv, startts, sclhandle, ierr)

Parameters

opid id of the operation; must be one of the predefined constants in enum _VT_OpTypes

commid numeric ID for the communicator; see VT_log_sendmsg() for valid numbers

root rank of the root process in the communicator (ignored for operations without root, must still be valid, though)

numprocs the number of processes in the communicator

bsend bytes sent by process

brecv bytes received by process

startts the start time of the operation (as returned by VT_timestamp())

sclhandle handle as defined by VT_scldef, or VT_NOSCL

Return values

Returns error code.

 

enum _VT_OpTypes

These are operation IDs that can be passed to VT_log_op().

Their representation in the trace file matches that of the equivalent MPI operation.

User-defined operations are not supported.

Enumerator Description

VT_OP_INVALID

Undefined operation, should not be passed to Intel® Trace Collector

VT_OP_COUNT

Number of predefined operations

VT_OP_BARRIER

VT_OP_IBARRIER

VT_OP_BCAST

VT_OP_IBCAST

VT_OP_GATHER

VT_OP_IGATHER

VT_OP_GATHERV

VT_OP_IGATHERV

VT_OP_SCATTER

VT_OP_ISCATTER

VT_OP_SCATTERV

VT_OP_ISCATTERV

VT_OP_ALLGATHER

VT_OP_IALLGATHER

VT_OP_ALLGATHERV

VT_OP_IALLGATHERV

VT_OP_ALLTOALL

VT_OP_IALLTOALL

VT_OP_ALLTOALLV

VT_OP_IALLTOALLV

VT_OP_ALLTOALLW

VT_OP_IALLTOALLW

VT_OP_NEIGHBOR_ALLGATHER

VT_OP_INEIGHBOR_ALLGATHER

VT_OP_NEIGHBOR_ALLGATHERV

VT_OP_INEIGHBOR_ALLGATHERV

VT_OP_NEIGHBOR_ALLTOALL

VT_OP_INEIGHBOR_ALLTOALL

VT_OP_NEIGHBOR_ALLTOALLV

VT_OP_INEIGHBOR_ALLTOALLV

VT_OP_NEIGHBOR_ALLTOALLW

VT_OP_INEIGHBOR_ALLTOALLW

VT_OP_REDUCE

VT_OP_REDUCE_LOCAL

VT_OP_IREDUCE

VT_OP_ALLREDUCE

VT_OP_IALLREDUCE

VT_OP_REDUCE_SCATTER

VT_OP_REDUCE_SCATTER_BLOCK

VT_OP_IREDUCE_SCATTER

VT_OP_IREDUCE_SCATTER_BLOCK

VT_OP_SCAN

VT_OP_ISCAN

VT_OP_EXSCAN

VT_OP_IEXSCAN

MPI operation representations

 

Having a duration also may introduce the problem of having overlapping operations, which has to be taken care of with the following two calls.

VT_begin_unordered

int VT_begin_unordered(void)

Description

Starts a period with out-of-order events.

Most API functions log events with just one time stamp which is taken when the event is logged. That guarantees strict chronological order of the events.

VT_log_msgevent() and VT_log_opevent() are logged when the event has finished with a start time taken earlier with VT_timestamp(). This can break the chronological order, for example, like in the following two examples:

t1: VT_timestamp() "start message"
t2: VT_end() "leave function"
t3: VT_log_msgevent( t1 ) "finish message"
t1: VT_timestamp() "start first message"
t2: VT_timestamp() "start second message"
t3: VT_log_msgevent( t1 ) "finish first message"
t4: VT_log_msgevent( t2 ) "finish second message"

In other words, it is okay to just log a complex event if and only if no other event is logged between its start and end in this thread. "logged" in this context includes other complex events that are logged later, but with a start time between the other events start and end time.

In all other cases you have to alert Intel® Trace Collector of the fact that out-of-order events will follow by calling VT_begin_unordered() before and VT_end_unordered() after these events. When writing the events into the trace file Intel® Trace Collector increases a counter per thread when it sees a VT_begin_unordered() and decrease it at a VT_end_unordered(). Events are remembered and sorted until the counter reaches zero, or till the end of the data.

This means that:

  • unordered periods can be nested

  • it is not necessary to close each unordered period at the end of the trace

  • but not closing them properly in the middle of a trace will force Intel® Trace Collector to use a lot more memory when writing the trace (proportional to the number of events till the end of the trace).

Fortran

VTBEGINUNORDERED(ierr)

 

VT_end_unordered

int VT_end_unordered (void)

Description

Close a period with out-of-order events that was started with VT_begin_unordered().

Fortran

VTENDUNORDERED(ierr)

Для получения подробной информации о возможностях оптимизации компилятора обратитесь к нашему Уведомлению об оптимизации.