IMB Allreduce and Reduce message size confusion

IMB Allreduce and Reduce message size confusion

Portrait de Pavel Mezentsev
Pavel Mezentsev
dim, 16/09/2012 - 09:08

I've been doing some tuning and noticed a confusing thing.

I've tried different types of Allreduce and Reduce for particular message size and noticed that in the output my setting influenced the message size 4 times bigger than what I've specified. For exaple if I say use this type for message size 128 actually message size 512 would be affected. For other collectives everything works as expected.

Documentation states that MINMSGLOG for Allreduce and Reduce is sizeof(float) which is 4 in my case instead of 1 for other cases by default. So perhaps the actual message size starts from 1 byte but the value printed is just 4 times bigger. Is it possible?

RSS Haut
4 posts / 0 nouveau(x)
Dernière contribution
Reportez-vous à notre Notice d'optimisation pour plus d'informations sur les choix et l'optimisation des performances dans les produits logiciels Intel.
Portrait de James Tullos (Intel)
James Tullos (Intel)
ven, 21/09/2012 - 07:26

Hi Pavel,

What command line are you using? If I use

mpirun -n 2 ./IMB-MPI1 -msglog 4 allreduce reduce sendrecv

I get a maximum message size of 16 (2^4) for each of the operations.

Sincerely,
James Tullos
Technical Consulting Engineer
Intel® Cluster Tools

Haut
Portrait de Pavel Mezentsev
Pavel Mezentsev
dim, 23/09/2012 - 14:51

Hello, James.

My command looks like this:
mpiexec.hydra -np $ NP ... ./IMB-MPI1 -npmin $NP -mem 4G $COLL

The maximum size is the same, but Allreduce skips sizes 1 and 2, which should be fine according to the documentation.

I'm doing some FCA testing with OpenMPI and in particular I'm trying to tune the FCA parameters in the fca_mpi_spec.ini file. That's where I got confused. For example I set the following rules:
[rule-allreduce-2]
msg_size_min = 64
msg_size_max = 64
comm_size_min = 80
comm_size_max = 80
offload_type = none
fca_enable = 1

## Sample bcast rules

[rule-bcast-1]
msg_size_min = 64
msg_size_max = 64
comm_size_min = 80
comm_size_max = 80
offload_type = none
fca_enable = 1

And here are the results I get:

#----------------------------------------------------------------
# Benchmarking Allreduce
# #processes = 80
#----------------------------------------------------------------
#bytes #repetitions t_min[usec] t_max[usec] t_avg[usec]
0 1000 0.02 0.02 0.02
4 1000 3.29 3.30 3.30
8 1000 3.24 3.24 3.24
16 1000 3.26 3.26 3.26
32 1000 3.34 3.35 3.34
64 1000 4.11 4.12 4.12
128 1000 4.35 4.35 4.35
256 1000 24.48 24.50 24.49
512 1000 6.42 6.43 6.42
1024 1000 8.16 8.16 8.16
2048 1000 13.55 13.56 13.55
........
#----------------------------------------------------------------
# Benchmarking Bcast
# #processes = 80
#----------------------------------------------------------------
#bytes #repetitions t_min[usec] t_max[usec] t_avg[usec]
0 1000 0.02 0.02 0.02
1 1000 2.60 2.61 2.61
2 1000 1.55 1.57 1.56
4 1000 1.51 1.52 1.51
8 1000 1.50 1.51 1.51
16 1000 1.50 1.51 1.51
32 1000 1.54 1.56 1.55
64 1000 7.91 7.95 7.93
128 1000 1.72 1.73 1.73
256 1000 1.74 1.75 1.74
512 1000 1.75 1.77 1.76
1024 1000 1.95 1.96 1.95
2048 1000 2.92 2.93 2.93
.......

In case of Allreduce the affected message size is four times bigger than the one specified in the FCA rules.

I'm trying to determine why the results are so weird. Is there a bug in IMB or in FCA or in OpenMPI or it's just something that I don't understand and everything should be the way it is :)

Haut
Portrait de James Tullos (Intel)
James Tullos (Intel)
lun, 24/09/2012 - 06:54

Hi Pavel,

I think the problem may be a difference in how "message size" is defined. For the Intel® MPI Benchmarks (and the Intel® MPI Library), the message size is defined by the number of bytes in the message. It is possible that the message size defined in fca_mpi_spec.ini is the number of elements. If so, the allreduce benchmark would show a difference at 4 times the message size. If you are using the standard code for IMB, all reduction operations use MPI_FLOAT (4 bytes), and everything else uses MPI_BYTE (1 byte). I would check with the Open MPI team about this.

Sincerely,
James Tullos
Technical Consulting Engineer
Intel® Cluster Tools

Haut

Connectez-vous pour laisser un commentaire.