Update: either with pinning, or with I_MPI_PIN=disable, there is the same problem: a job that succesfully runs interactively on two nodes, fails when run under Torque, on the very same nodes, with following error message:
Fatal error in PMPI_Bcast: Message truncated, error stack:
PMPI_Bcast(1920)..................: MPI_Bcast(buf=0x7fff86db0ac0, count=1, MPI_LONG, root=0, MPI_COMM_WORLD) failed
MPIR_Bcast(1236)..................:
MPIR_Bcast_Shum_ring(1039)........:
MPIDI_CH3U_Receive_data_found(129): Message from rank 3 and tag 2 truncated; 532 bytes received but buffer size is 8
[1:n008] unexpected disconnect completion event from [4:n189]
Assertion failed in file ../../dapl_module_util.c at line 2682: 0
~
I'm at loss as to what might cause the difference..
--
Grigory Shamov
How to pin Intel MPI processes within Torque cpusets? Set domain issues
Hi,
I think I have a problem with process pinning, for older version of Intel MPI (4.0.1). The version cannot be changed because it is bundled with the user's application ( Accelrys Material Studio) and there are tons of scripts surrounding it. The code works when started interactively, but when run under the Torque batch system, there are following messages:
[6] MPI startup(): set domain {10,11} fails on node XXX.local
[5] MPI startup(): set domain {9} fails on node XXX.local
[7] MPI startup(): set domain {10,11} fails on node XXX.local
[4] MPI startup(): set domain {9} fails on node XXX.local
The code then fails when is run accross the nodes, or runs slow within a single node. I can run the same code, same data interactibvely accross the nodes, and I dont see teh "set domain" messages.
Our site uses Torque cpusets. So I suspect the difference between running interactively or from a batch script is the cpusets and pinning of the processes.
First question: am I correct? What do these "set domain fails" messages really mean?). Torque gives the list of CPU cores allocated for the job in its cpuset: /dev/cpuset/torque/JOB_ID/cpus will contain something like "8-11" or "0-7". I have tried to pass it to Intel MPI as follows:
range=`cat /dev/cpuset/torque/$PBS_JOBID/cpus`
export I_MPI_PIN=enable
export I_MPI_PIN_PROCS=$range
[... RunMatServer.sh starts ...]
It seems to pin cores to something. And I dont get the "set domain" messages anymore. The second question is, is that a right/correct way to interface Torque cpusets to IntelMPI jobs?
--
Grigory Shamov
University of Manitoba / Westgrid