Loading...

• Next Thread Prev Thread

• Tags

  InfiniBand MPI benchmark

• Posts   Search Threads

• jriocaton.es June 29, 2009 12:49 AM PDT    Infiniband-Intel MPI Performance Dear colleagues, new doubts testing infiniband performance. The results are the same for both of them, gigabit and infiniband. Could you please help me ? Thanks --> Infiniband [root@cn035 ~]# time -p /soft/intel/impi/3.2.1.009/bin64/mpiexec -np 16 -env I_MPI_DEBUG 2 -env I_MPI_DEVICE rdma /home/c/OMB-3.1.1/osu_mbw_mr [0] MPI startup(): DAPL provider ofa-v2-ib0 specified in DAPL configuration file /etc/dat.conf [8] MPI startup(): DAPL provider ofa-v2-ib0 specified in DAPL configuration file /etc/dat.conf [1] MPI startup(): DAPL provider ofa-v2-ib0 specified in DAPL configuration file /etc/dat.conf [7] MPI startup(): DAPL provider ofa-v2-ib0 specified in DAPL configuration file /etc/dat.conf [2] MPI startup(): DAPL provider ofa-v2-ib0 specified in DAPL configuration file /etc/dat.conf [5] MPI startup(): DAPL provider ofa-v2-ib0 specified in DAPL configuration file /etc/dat.conf [11] MPI startup(): DAPL provider ofa-v2-ib0 specified in DAPL configuration file /etc/dat.conf [3] MPI startup(): DAPL provider ofa-v2-ib0 specified in DAPL configuration file /etc/dat.conf [15] MPI startup(): DAPL provider ofa-v2-ib0 specified in DAPL configuration file /etc/dat.conf [13] MPI startup(): DAPL provider ofa-v2-ib0 specified in DAPL configuration file /etc/dat.conf [4] MPI startup(): DAPL provider ofa-v2-ib0 specified in DAPL configuration file /etc/dat.conf [6] MPI startup(): DAPL provider ofa-v2-ib0 specified in DAPL configuration file /etc/dat.conf [14] MPI startup(): DAPL provider ofa-v2-ib0 specified in DAPL configuration file /etc/dat.conf [9] MPI startup(): DAPL provider ofa-v2-ib0 specified in DAPL configuration file /etc/dat.conf [12] MPI startup(): DAPL provider ofa-v2-ib0 specified in DAPL configuration file /etc/dat.conf [10] MPI startup(): DAPL provider ofa-v2-ib0 specified in DAPL configuration file /etc/dat.conf [0] MPI startup(): RDMA data transfer mode [2] MPI startup(): RDMA data transfer mode [5] MPI startup(): RDMA data transfer mode [1] MPI startup(): RDMA data transfer mode [4] MPI startup(): RDMA data transfer mode [6] MPI startup(): RDMA data transfer mode [10] MPI startup(): RDMA data transfer mode [11] MPI startup(): RDMA data transfer mode [12] MPI startup(): RDMA data transfer mode [9] MPI startup(): RDMA data transfer mode [13] MPI startup(): RDMA data transfer mode [7] MPI startup(): RDMA data transfer mode [3] MPI startup(): RDMA data transfer mode [8] MPI startup(): RDMA data transfer mode [15] MPI startup(): RDMA data transfer mode [14] MPI startup(): RDMA data transfer mode [8] MPI Startup(): process is pinned to CPU00 on node cn038 [6] MPI Startup(): process is pinned to CPU03 on node cn035 [9] MPI Startup(): process is pinned to CPU04 on node cn038 [10] MPI Startup(): process is pinned to CPU02 on node cn038 [7] MPI Startup(): process is pinned to CPU07 on node cn035 [11] MPI Startup(): process is pinned to CPU06 on node cn038 [0] MPI Startup(): process is pinned to CPU00 on node cn035 # OSU MPI Multiple Bandwidth / Message Rate Test v3.1.1 # [ pairs: 8 ] [ window size: 64 ] # Size MB/s Messages/s [12] MPI Startup(): process is pinned to CPU01 on node cn038 [5] MPI Startup(): process is pinned to CPU05 on node cn035 [1] MPI Startup(): process is pinned to CPU04 on node cn035 [13] MPI Startup(): process is pinned to CPU05 on node cn038 [14] MPI Startup(): process is pinned to CPU03 on node cn038 [4] MPI Startup(): process is pinned to CPU01 on node cn035 [3] MPI Startup(): process is pinned to CPU06 on node cn035 [2] MPI Startup(): process is pinned to CPU02 on node cn035 [15] MPI Startup(): process is pinned to CPU07 on node cn038 1 8.60 8600599.34 2 16.82 8409961.42 4 31.48 7870853.42 8 33.62 4202594.27 16 102.57 6410590.31 32 236.60 7393643.13 64 473.13 7392625.04 128 712.43 5565880.43 256 556.57 2174094.57 512 779.38 1522217.56 1024 1011.18 987485.01 2048 1129.40 551462.61 4096 1241.51 303104.13 8192 1332.10 162610.42 16384 1323.28 80766.63 32768 1416.24 43220.23 65536 1419.74 21663.56 131072 1428.36 10897.51 262144 1434.37 5471.68 524288 1434.59 2736.27 1048576 1433.45 1367.05 2097152 1404.73 669.83 4194304 1404.14 334.77 --> Gigabit [root@cn035 ~]# time -p /soft/intel/impi/3.2.1.009/bin64/mpiexec -np 16 -env I_MPI_DEBUG 2 /home/c/OMB-3.1.1/osu_mbw_mr [6] MPI startup(): DAPL provider ofa-v2-ib0 specified in DAPL configuration file /etc/dat.conf [5] MPI startup(): DAPL provider ofa-v2-ib0 specified in DAPL configuration file /etc/dat.conf [0] MPI startup(): DAPL provider ofa-v2-ib0 specified in DAPL configuration file /etc/dat.conf [1] MPI startup(): DAPL provider ofa-v2-ib0 specified in DAPL configuration file /etc/dat.conf [10] MPI startup(): DAPL provider ofa-v2-ib0 specified in DAPL configuration file /etc/dat.conf [7] MPI startup(): DAPL provider ofa-v2-ib0 specified in DAPL configuration file /etc/dat.conf [4] MPI startup(): DAPL provider ofa-v2-ib0 specified in DAPL configuration file /etc/dat.conf [2] MPI startup(): DAPL provider ofa-v2-ib0 specified in DAPL configuration file /etc/dat.conf [3] MPI startup(): DAPL provider ofa-v2-ib0 specified in DAPL configuration file /etc/dat.conf [13] MPI startup(): DAPL provider ofa-v2-ib0 specified in DAPL configuration file /etc/dat.conf [8] MPI startup(): DAPL provider ofa-v2-ib0 specified in DAPL configuration file /etc/dat.conf [9] MPI startup(): DAPL provider ofa-v2-ib0 specified in DAPL configuration file /etc/dat.conf [11] MPI startup(): DAPL provider ofa-v2-ib0 specified in DAPL configuration file /etc/dat.conf [12] MPI startup(): DAPL provider ofa-v2-ib0 specified in DAPL configuration file /etc/dat.conf [14] MPI startup(): DAPL provider ofa-v2-ib0 specified in DAPL configuration file /etc/dat.conf [15] MPI startup(): DAPL provider ofa-v2-ib0 specified in DAPL configuration file /etc/dat.conf [0] MPI startup(): RDMA, shared memory, and socket data transfer modes [1] MPI startup(): RDMA, shared memory, and socket data transfer modes [2] MPI startup(): RDMA, shared memory, and socket data transfer modes [3] MPI startup(): RDMA, shared memory, and socket data transfer modes [4] MPI startup(): RDMA, shared memory, and socket data transfer modes [5] MPI startup(): RDMA, shared memory, and socket data transfer modes [7] MPI startup(): RDMA, shared memory, and socket data transfer modes [8] MPI startup(): RDMA, shared memory, and socket data transfer modes [9] MPI startup(): RDMA, shared memory, and socket data transfer modes [10] MPI startup(): RDMA, shared memory, and socket data transfer modes [11] MPI startup(): RDMA, shared memory, and socket data transfer modes [12] MPI startup(): RDMA, shared memory, and socket data transfer modes [13] MPI startup(): RDMA, shared memory, and socket data transfer modes [6] MPI startup(): RDMA, shared memory, and socket data transfer modes [14] MPI startup(): RDMA, shared memory, and socket data transfer modes [15] MPI startup(): RDMA, shared memory, and socket data transfer modes [1] MPI Startup(): process is pinned to CPU04 on node cn035 [0] MPI Startup(): process is pinned to CPU00 on node cn035 # OSU MPI Multiple Bandwidth / Message Rate Test v3.1.1 # [ pairs: 8 ] [ window size: 64 ] # Size MB/s Messages/s [2] MPI Startup(): process is pinned to CPU02 on node cn035 [7] MPI Startup(): process is pinned to CPU07 on node cn035 [11] MPI Startup(): process is pinned to CPU06 on node cn038 [3] MPI Startup(): process is pinned to CPU06 on node cn035 [6] MPI Startup(): process is pinned to CPU03 on node cn035 [10] MPI Startup(): process is pinned to CPU02 on node cn038 [13] MPI Startup(): process is pinned to CPU05 on node cn038 [5] MPI Startup(): process is pinned to CPU05 on node cn035 [14] MPI Startup(): process is pinned to CPU03 on node cn038 [4] MPI Startup(): process is pinned to CPU01 on node cn035 [12] MPI Startup(): [8] MPI Startup(): process is pinned to CPU00 on node cn038 process is pinned to CPU01 on node cn038 [9] MPI Startup(): process is pinned to CPU04 on node cn038 [15] MPI Startup(): process is pinned to CPU07 on node cn038 1 8.44 8436060.84 2 15.39 7695694.85 4 29.90 7474447.98 8 66.10 8262094.68 16 102.29 6392842.49 32 236.52 7391352.82 64 473.65 7400777.64 128 712.73 5568189.51 256 555.32 2169219.22 512 777.95 1519428.06 1024 1005.73 982160.29 2048 1131.15 552319.28 4096 1240.71 302906.61 8192 1328.67 162191.63 16384 1372.56 83774.33 32768 1407.69 42959.17 65536 1421.51 21690.46 131072 1427.63 10891.95 262144 1433.66 5468.99 524288 1434.69 2736.46 1048576 1433.19 1366.80 2097152 1404.70 669.81 4194304 1404.09 334.76 --> /etc/dat.conf [root@cn035 ~]# more /etc/dat.conf ofa-v2-ib0 u2.0 nonthreadsafe default libdaplofa.so.2 dapl.2.0 "ib0 0" "" ofa-v2-ib1 u2.0 nonthreadsafe default libdaplofa.so.2 dapl.2.0 "ib1 0" "" ofa-v2-mthca0-1 u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 "mthca0 1" "" ofa-v2-mthca0-2 u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 "mthca0 2" "" ofa-v2-mlx4_0-1 u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 "mlx4_0 1" "" ofa-v2-mlx4_0-2 u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 "mlx4_0 2" "" ofa-v2-ipath0-1 u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 "ipath0 1" "" ofa-v2-ipath0-2 u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 "ipath0 2" "" ofa-v2-ehca0-1 u2.0 nonthreadsafe default libdaploscm.so.2 dapl.2.0 "ehca0 1" "" ofa-v2-iwarp u2.0 nonthreadsafe default libdaplofa.so.2 dapl.2.0 "eth2 0" "" OpenIB-cma u1.2 nonthreadsafe default libdaplcma.so.1 dapl.1.2 "ib0 0" "" OpenIB-cma-1 u1.2 nonthreadsafe default libdaplcma.so.1 dapl.1.2 "ib1 0" "" OpenIB-mthca0-1 u1.2 nonthreadsafe default libdaplscm.so.1 dapl.1.2 "mthca0 1" "" OpenIB-mthca0-2 u1.2 nonthreadsafe default libdaplscm.so.1 dapl.1.2 "mthca0 2" "" OpenIB-mlx4_0-1 u1.2 nonthreadsafe default libdaplscm.so.1 dapl.1.2 "mlx4_0 1" "" OpenIB-mlx4_0-2 u1.2 nonthreadsafe default libdaplscm.so.1 dapl.1.2 "mlx4_0 2" "" OpenIB-ipath0-1 u1.2 nonthreadsafe default libdaplscm.so.1 dapl.1.2 "ipath0 1" "" OpenIB-ipath0-2 u1.2 nonthreadsafe default libdaplscm.so.1 dapl.1.2 "ipath0 2" "" OpenIB-ehca0-1 u1.2 nonthreadsafe default libdaplscm.so.1 dapl.1.2 "ehca0 1" "" OpenIB-iwarp u1.2 nonthreadsafe default libdaplcma.so.1 dapl.1.2 "eth2 0" "" TimP (Intel) June 29, 2009 7:50 AM PDT Rate   #1 Re: Infiniband-Intel MPI Performance Accordiing to what you have posted, your installation defaults to rdsm, as expected for the version you have chosen.  If you wish to test a non-rdma device path, you must specify it. Gergana Slavova (Intel) June 29, 2009 12:56 PM PDT Rate   #2 Re: Infiniband-Intel MPI Performance Hello Julio, As Tim mentions, you're actually running the application over InfiniBand* in both instances. A few releases ago, Intel® MPI Library had changed the defaults to use the fastest available network on the cluster at startup (which would be InfiniBand, in your case). Below, it seems like you specify the RDMA device in your IB run, but don't specify a device in your GigE run (which would default to IB again): Quoting - jriocaton.es --> Infiniband [root@cn035 ~]# time -p /soft/intel/impi/3.2.1.009/bin64/mpiexec -np 16 -env I_MPI_DEBUG 2 -env I_MPI_DEVICE rdma /home/c/OMB-3.1.1/osu_mbw_mr [0] MPI startup(): DAPL provider ofa-v2-ib0 specified in DAPL configuration file /etc/dat.conf ... [0] MPI startup(): RDMA data transfer mode --> Gigabit [root@cn035 ~]# time -p /soft/intel/impi/3.2.1.009/bin64/mpiexec -np 16 -env I_MPI_DEBUG 2 /home/c/OMB-3.1.1/osu_mbw_mr [0] MPI startup(): DAPL provider ofa-v2-ib0 specified in DAPL configuration file /etc/dat.conf ... [0] MPI startup(): RDMA, shared memory, and socket data transfer modes Since the default is the RDSSM device, your two runs are identical. If you'd like to run over GigE, you need to specify the "sock" device. Alternatively, you can use "ssm" for GigE communication between the nodes, and shared memory within a node. That means your command line would look like this: # time -p /soft/intel/impi/3.2.1.009/bin64/mpiexec -np 16 -env I_MPI_DEBUG 2 -env I_MPI_DEVICE ssm /home/c/OMB-3.1.1/osu_mbw_mr I hope this helps. Regards, ~Gergana Gergana Slavova Technical Consulting Engineer Intel® Cluster Tools E-mail: gergana.s.slavova_at_intel.com jriocaton.es July 3, 2009 1:08 AM PDT Rate   #3 Reply To #2 Re: Infiniband-Intel MPI Performance I am sorry for the delay, Ive been out of the office. Thanks a lot for your help, Ill try your recomendations

• = Best Answer

  Next Thread Prev Thread

Intel Software Network Forums Statistics

16,377 users have contributed to 46,364 threads and 164,041 posts to date.

For more complete information about compiler optimizations, see our Optimization Notice.