We have several new IBM iDataplexes. Some of our codes compiled with Intel 12.1 with INTEL-MPI-4.0.3 would sometimes fail with this error:

"APPLICATION TERMINATED WITH THE EXIT STRING: Hangup (signal 1)"

I can consistenly replicate this error with the Intel IMB-MPI1.4.0.3 benchmark system on two nodes (32 cores).

The error above happens in the Allgatherv benchmark using 32 processes after te 8192 byte size messages (see below).

*BUT*, if I were to JUST RUN an Allgatherv benchmark, it works with no problems. It appears a previous MPI funciton call is setting the system in some state to cause Allfatherv to fail.

#----------------------------------------------------------------
# Benchmarking Allgatherv
# #processes = 32
#----------------------------------------------------------------
#bytes #repetitions t_min[usec] t_max[usec] t_avg[usec]
0 1000 0.18 0.19 0.18
1 1000 55.13 55.16 55.14
2 1000 55.58 55.60 55.59
4 1000 55.42 55.44 55.43
8 1000 54.99 55.02 55.01
16 1000 56.93 56.95 56.94
32 1000 60.37 60.37 60.37
64 1000 60.45 60.45 60.45
128 1000 59.13 59.14 59.13
256 1000 152.55 152.59 152.57
512 1000 152.85 152.90 152.88
1024 1000 92.38 92.39 92.39
2048 1000 198.94 199.08 198.98
4096 1000 244.89 245.09 244.97
8192 1000 323.58 323.74 323.70
APPLICATION TERMINATED WITH THE EXIT STRING: Hangup (signal 1)