Forum Jump

Select Group :
Select Forum :
Sorted By :
Sort Order :
From The :
 
Thread Tools  Search this thread 
jriocaton.es
Total Points:
185
Status Points:
135
Green Belt
June 29, 2009 12:49 AM PDT
Infiniband-Intel MPI Performance
Dear colleagues,

new doubts testing infiniband performance. The results are the same for both of them, gigabit and infiniband. Could you please help me ? Thanks

--> Infiniband

[root@cn035 ~]# time -p /soft/intel/impi/3.2.1.009/bin64/mpiexec -np 16 -env I_MPI_DEBUG 2 -env I_MPI_DEVICE rdma /home/c/OMB-3.1.1/osu_mbw_mr
[0] MPI startup(): DAPL provider ofa-v2-ib0 specified in DAPL configuration file /etc/dat.conf
[8] MPI startup(): DAPL provider ofa-v2-ib0 specified in DAPL configuration file /etc/dat.conf
[1] MPI startup(): DAPL provider ofa-v2-ib0 specified in DAPL configuration file /etc/dat.conf
[7] MPI startup(): DAPL provider ofa-v2-ib0 specified in DAPL configuration file /etc/dat.conf
[2] MPI startup(): DAPL provider ofa-v2-ib0 specified in DAPL configuration file /etc/dat.conf
[5] MPI startup(): DAPL provider ofa-v2-ib0 specified in DAPL configuration file /etc/dat.conf
[11] MPI startup(): DAPL provider ofa-v2-ib0 specified in DAPL configuration file /etc/dat.conf
[3] MPI startup(): DAPL provider ofa-v2-ib0 specified in DAPL configuration file /etc/dat.conf
[15] MPI startup(): DAPL provider ofa-v2-ib0 specified in DAPL configuration file /etc/dat.conf
[13] MPI startup(): DAPL provider ofa-v2-ib0 specified in DAPL configuration file /etc/dat.conf
[4] MPI startup(): DAPL provider ofa-v2-ib0 specified in DAPL configuration file /etc/dat.conf
[6] MPI startup(): DAPL provider ofa-v2-ib0 specified in DAPL configuration file /etc/dat.conf
[14] MPI startup(): DAPL provider ofa-v2-ib0 specified in DAPL configuration file /etc/dat.conf
[9] MPI startup(): DAPL provider ofa-v2-ib0 specified in DAPL configuration file /etc/dat.conf
[12] MPI startup(): DAPL provider ofa-v2-ib0 specified in DAPL configuration file /etc/dat.conf
[10] MPI startup(): DAPL provider ofa-v2-ib0 specified in DAPL configuration file /etc/dat.conf
[0] MPI startup(): RDMA data transfer mode
[2] MPI startup(): RDMA data transfer mode
[5] MPI startup(): RDMA data transfer mode
[1] MPI startup(): RDMA data transfer mode
[4] MPI startup(): RDMA data transfer mode
[6] MPI startup(): RDMA data transfer mode
[10] MPI startup(): RDMA data transfer mode
[11] MPI startup(): RDMA data transfer mode
[12] MPI startup(): RDMA data transfer mode
[9] MPI startup(): RDMA data transfer mode
[13] MPI startup(): RDMA data transfer mode
[7] MPI startup(): RDMA data transfer mode
[3] MPI startup(): RDMA data transfer mode
[8] MPI startup(): RDMA data transfer mode
[15] MPI startup(): RDMA data transfer mode
[14] MPI startup(): RDMA data transfer mode
[8] MPI Startup(): process is pinned to CPU00 on node cn038
[6] MPI Startup(): process is pinned to CPU03 on node cn035
[9] MPI Startup(): process is pinned to CPU04 on node cn038
[10] MPI Startup(): process is pinned to CPU02 on node cn038
[7] MPI Startup(): process is pinned to CPU07 on node cn035
[11] MPI Startup(): process is pinned to CPU06 on node cn038
[0] MPI Startup(): process is pinned to CPU00 on node cn035
# OSU MPI Multiple Bandwidth / Message Rate Test v3.1.1
# [ pairs: 8 ] [ window size: 64 ]
# Size MB/s Messages/s
[12] MPI Startup(): process is pinned to CPU01 on node cn038
[5] MPI Startup(): process is pinned to CPU05 on node cn035
[1] MPI Startup(): process is pinned to CPU04 on node cn035
[13] MPI Startup(): process is pinned to CPU05 on node cn038
[14] MPI Startup(): process is pinned to CPU03 on node cn038
[4] MPI Startup(): process is pinned to CPU01 on node cn035
[3] MPI Startup(): process is pinned to CPU06 on node cn035
[2] MPI Startup(): process is pinned to CPU02 on node cn035
[15] MPI Startup(): process is pinned to CPU07 on node cn038
1 8.60 8600599.34
2 16.82 8409961.42
4 31.48 7870853.42
8 33.62 4202594.27
16 102.57 6410590.31
32 236.60 7393643.13
64 473.13 7392625.04
128 712.43 5565880.43
256 556.57 2174094.57
512 779.38 1522217.56
1024 1011.18 987485.01
2048 1129.40 551462.61
4096 1241.51 303104.13
8192 1332.10 162610.42
16384 1323.28 80766.63
32768 1416.24 43220.23
65536 1419.74 21663.56
131072 1428.36 10897.51
262144 1434.37 5471.68
524288 1434.59 2736.27
1048576 1433.45 1367.05
2097152 1404.73 669.83
4194304 1404.14 334.77

--> Gigabit

[root@cn035 ~]# time -p /soft/intel/impi/3.2.1.009/bin64/mpiexec -np 16 -env I_MPI_DEBUG 2 /home/c/OMB-3.1.1/osu_mbw_mr
[6] MPI startup(): DAPL provider ofa-v2-ib0 specified in DAPL configuration file /etc/dat.conf
[5] MPI startup(): DAPL provider ofa-v2-ib0 specified in DAPL configuration file /etc/dat.conf
[0] MPI startup(): DAPL provider ofa-v2-ib0 specified in DAPL configuration file /etc/dat.conf
[1] MPI startup(): DAPL provider ofa-v2-ib0 specified in DAPL configuration file /etc/dat.conf
[10] MPI startup(): DAPL provider ofa-v2-ib0 specified in DAPL configuration file /etc/dat.conf
[7] MPI startup(): DAPL provider ofa-v2-ib0 specified in DAPL configuration file /etc/dat.conf
[4] MPI startup(): DAPL provider ofa-v2-ib0 specified in DAPL configuration file /etc/dat.conf
[2] MPI startup(): DAPL provider ofa-v2-ib0 specified in DAPL configuration file /etc/dat.conf
[3] MPI startup(): DAPL provider ofa-v2-ib0 specified in DAPL configuration file /etc/dat.conf
[13] MPI startup(): DAPL provider ofa-v2-ib0 specified in DAPL configuration file /etc/dat.conf
[8] MPI startup(): DAPL provider ofa-v2-ib0 specified in DAPL configuration file /etc/dat.conf
[9] MPI startup(): DAPL provider ofa-v2-ib0 specified in DAPL configuration file /etc/dat.conf
[11] MPI startup(): DAPL provider ofa-v2-ib0 specified in DAPL configuration file /etc/dat.conf
[12] MPI startup(): DAPL provider ofa-v2-ib0 specified in DAPL configuration file /etc/dat.conf
[14] MPI startup(): DAPL provider ofa-v2-ib0 specified in DAPL configuration file /etc/dat.conf
[15] MPI startup(): DAPL provider ofa-v2-ib0 specified in DAPL configuration file /etc/dat.conf
[0] MPI startup(): RDMA, shared memory, and socket data transfer modes
[1] MPI startup(): RDMA, shared memory, and socket data transfer modes
[2] MPI startup(): RDMA, shared memory, and socket data transfer modes
[3] MPI startup(): RDMA, shared memory, and socket data transfer modes
[4] MPI startup(): RDMA, shared memory, and socket data transfer modes
[5] MPI startup(): RDMA, shared memory, and socket data transfer modes
[7] MPI startup(): RDMA, shared memory, and socket data transfer modes
[8] MPI startup(): RDMA, shared memory, and socket data transfer modes
[9] MPI startup(): RDMA, shared memory, and socket data transfer modes
[10] MPI startup(): RDMA, shared memory, and socket data transfer modes
[11] MPI startup(): RDMA, shared memory, and socket data transfer modes
[12] MPI startup(): RDMA, shared memory, and socket data transfer modes
[13] MPI startup(): RDMA, shared memory, and socket data transfer modes
[6] MPI startup(): RDMA, shared memory, and socket data transfer modes
[14] MPI startup(): RDMA, shared memory, and socket data transfer modes
[15] MPI startup(): RDMA, shared memory, and socket data transfer modes
[1] MPI Startup(): process is pinned to CPU04 on node cn035
[0] MPI Startup(): process is pinned to CPU00 on node cn035
# OSU MPI Multiple Bandwidth / Message Rate Test v3.1.1
# [ pairs: 8 ] [ window size: 64 ]
# Size MB/s Messages/s
[2] MPI Startup(): process is pinned to CPU02 on node cn035
[7] MPI Startup(): process is pinned to CPU07 on node cn035
[11] MPI Startup(): process is pinned to CPU06 on node cn038
[3] MPI Startup(): process is pinned to CPU06 on node cn035
[6] MPI Startup(): process is pinned to CPU03 on node cn035
[10] MPI Startup(): process is pinned to CPU02 on node cn038
[13] MPI Startup(): process is pinned to CPU05 on node cn038
[5] MPI Startup(): process is pinned to CPU05 on node cn035
[14] MPI Startup(): process is pinned to CPU03 on node cn038
[4] MPI Startup(): process is pinned to CPU01 on node cn035
[12] MPI Startup(): [8] MPI Startup(): process is pinned to CPU00 on node cn038
process is pinned to CPU01 on node cn038
[9] MPI Startup(): process is pinned to CPU04 on node cn038
[15] MPI Startup(): process is pinned to CPU07 on node cn038
1 8.44 8436060.84
2 15.39 7695694.85
4 29.90 7474447.98
8 66.10 8262094.68
16 102.29 6392842.49
32 236.52 7391352.82
64 473.65 7400777.64
128 712.73 5568189.51
256 555.32 2169219.22
512 777.95 1519428.06
1024 1005.73 982160.29
2048 1131.15 552319.28
4096 1240.71 302906.61
8192 1328.67 162191.63
16384 1372.56 83774.33
32768 1407.69 42959.17
65536 1421.51 21690.46
131072 1427.63 10891.95
262144 1433.66 5468.99
524288 1434.69 2736.46
1048576 1433.19 1366.80
2097152 1404.70 669.81
4194304 1404.09 334.76

--> /etc/dat.conf
[root@cn035 ~]# more /etc/dat.conf
ofa-v2-ib0 u2.0 nonthreadsafe default libdaplofa.so.2 dapl.2.0 "ib0 0" ""
ofa-v2-ib1 u2.0 nonthreadsafe default libdaplofa.so.2 dapl.2.0 "ib1 0" ""
ofa-v2-mthca0-1 u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 "mthca0 1" ""
ofa-v2-mthca0-2 u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 "mthca0 2" ""
ofa-v2-mlx4_0-1 u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 "mlx4_0 1" ""
ofa-v2-mlx4_0-2 u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 "mlx4_0 2" ""
ofa-v2-ipath0-1 u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 "ipath0 1" ""
ofa-v2-ipath0-2 u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 "ipath0 2" ""
ofa-v2-ehca0-1 u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 "ehca0 1" ""
ofa-v2-iwarp u2.0 nonthreadsafe default libdaplofa.so.2 dapl.2.0 "eth2 0" ""
OpenIB-cma u1.2 nonthreadsafe default libdaplcma.so.1 dapl.1.2 "ib0 0" ""
OpenIB-cma-1 u1.2 nonthreadsafe default libdaplcma.so.1 dapl.1.2 "ib1 0" ""
OpenIB-mthca0-1 u1.2 nonthreadsafe default libdaplscm.so.1 dapl.1.2 "mthca0 1" ""
OpenIB-mthca0-2 u1.2 nonthreadsafe default libdaplscm.so.1 dapl.1.2 "mthca0 2" ""
OpenIB-mlx4_0-1 u1.2 nonthreadsafe default libdaplscm.so.1 dapl.1.2 "mlx4_0 1" ""
OpenIB-mlx4_0-2 u1.2 nonthreadsafe default libdaplscm.so.1 dapl.1.2 "mlx4_0 2" ""
OpenIB-ipath0-1 u1.2 nonthreadsafe default libdaplscm.so.1 dapl.1.2 "ipath0 1" ""
OpenIB-ipath0-2 u1.2 nonthreadsafe default libdaplscm.so.1 dapl.1.2 "ipath0 2" ""
OpenIB-ehca0-1 u1.2 nonthreadsafe default libdaplscm.so.1 dapl.1.2 "ehca0 1" ""
OpenIB-iwarp u1.2 nonthreadsafe default libdaplcma.so.1 dapl.1.2 "eth2 0" ""




tim18
Total Points:
66,397
Status Points:
66,397
Black Belt
June 29, 2009 7:50 AM PDT
Rate
 
#1
Accordiing to what you have posted, your installation defaults to rdsm, as expected for the version you have chosen.  If you wish to test a non-rdma device path, you must specify it.
Gergana Slavova (Intel)
Total Points:
4,274
Status Points:
3,774
Brown Belt
June 29, 2009 12:56 PM PDT
Rate
 
#2

Hello Julio,

As Tim mentions, you're actually running the application over InfiniBand* in both instances. A few releases ago, Intel® MPI Library had changed the defaults to use the fastest available network on the cluster at startup (which would be InfiniBand, in your case).

Below, it seems like you specify the RDMA device in your IB run, but don't specify a device in your GigE run (which would default to IB again):

Quoting - jriocaton.es
--> Infiniband

[root@cn035 ~]# time -p /soft/intel/impi/3.2.1.009/bin64/mpiexec -np 16 -env I_MPI_DEBUG 2 -env I_MPI_DEVICE rdma /home/c/OMB-3.1.1/osu_mbw_mr
[0] MPI startup(): DAPL provider ofa-v2-ib0 specified in DAPL configuration file /etc/dat.conf
...
[0] MPI startup(): RDMA data transfer mode

--> Gigabit

[root@cn035 ~]# time -p /soft/intel/impi/3.2.1.009/bin64/mpiexec -np 16 -env I_MPI_DEBUG 2 /home/c/OMB-3.1.1/osu_mbw_mr
[0] MPI startup(): DAPL provider ofa-v2-ib0 specified in DAPL configuration file /etc/dat.conf
...
[0] MPI startup(): RDMA, shared memory, and socket data transfer modes

Since the default is the RDSSM device, your two runs are identical. If you'd like to run over GigE, you need to specify the "sock" device. Alternatively, you can use "ssm" for GigE communication between the nodes, and shared memory within a node. That means your command line would look like this:

# time -p /soft/intel/impi/3.2.1.009/bin64/mpiexec -np 16 -env I_MPI_DEBUG 2 -env I_MPI_DEVICE ssm /home/c/OMB-3.1.1/osu_mbw_mr
I hope this helps.

Regards,
~Gergana


--------
Gergana Slavova
Technical Consulting Engineer
Intel® Cluster Tools
E-mail: gergana.s.slavova_at_intel.com
jriocaton.es
Total Points:
185
Status Points:
135
Green Belt
July 3, 2009 1:08 AM PDT
Rate
 
#3 Reply to #2
I am sorry for the delay, Ive been out of the office.

Thanks a lot for your help, Ill try your recomendations


Intel Software Network Forums Statistics

8285 users have contributed to 31229 threads and 99107 posts to date.
In the past 24 hours, we have 11 new thread(s) 45 new posts(s), and 61 new user(s).

In the past 3 days, the most popular thread for everyone has been comparison cilk++, openmp, pthreads first results The most posts were made to comparison cilk++, openmp, pthreads first results The post with the most views is Very amusing...  Escalated as

Please welcome our newest member tvinni