To avoid deadlock situations, you can use the following approaches:
Reorder MPI communication calls between processes.
Implement non-blocking calls.
Use MPI_Sendrecv or MPI_Sendrecv_replace.
Use the buffered mode.
The following code section leads to a deadlock in your original application:
... 52 /* waiting for a message that has not been sent */ 53 MPI_Recv( &recv, 1, MPI_CHAR, peer, 100, MPI_COMM_WORLD, &status ); 54 55 /* 56 * Too late, this code is not going to be reached. 57 * Beware, simply moving this code up would rely on 58 * buffering in the MPI. The correct solution is to 59 * use MPI_Isend() before the receive and MPI_Wait() 60 * afterwards. 61 */ 62 send = 0; 63 MPI_Send( &send, 1, MPI_CHAR, peer, 100, MPI_COMM_WORLD ); ...
To resolve the deadlock for the given example, you need to replace the MPI_Recv and MPI_Send calls with MPI_Sendrecv. Do the following:
Remove all lines from 52 to 61.
Replace the MPI_Recv function call (line 63) with the following MPI_Sendrecv call:
MPI_Sendrecv( &send, 1, MPI_CHAR, peer, 100, &recv, 1, MPI_CHAR, peer, 100, MPI_COMM_WORLD, &status );
Save this information into the file: MPI_Sendrecv.c.
Compile and run the modified application with the same parameters. The resulting output should look as follows:
 INFO: Writing tracefile MPI_Sendrecv.stf in /checking/global/deadlock/hard
 INFO: Error checking completed without finding any problems.
View the newly generated trace file with Intel® Trace Analyzer to make sure that the deadlock issue has been resolved:
As shown in the figure above, the deadlock problem no longer occurs, and both ranks successfully exchanged the messages.