Hello Julio,
As Tim mentions, you're actually running the application over InfiniBand* in both instances. A few releases ago, Intel® MPI Library had changed the defaults to use the fastest available network on the cluster at startup (which would be InfiniBand, in your case).
Below, it seems like you specify the RDMA device in your IB run, but don't specify a device in your GigE run (which would default to IB again):
--> Infiniband
[root@cn035 ~]# time -p /soft/intel/impi/3.2.1.009/bin64/mpiexec -np 16 -env I_MPI_DEBUG 2 -env I_MPI_DEVICE rdma /home/c/OMB-3.1.1/osu_mbw_mr
[0] MPI startup(): DAPL provider ofa-v2-ib0 specified in DAPL configuration file /etc/dat.conf
...
[0] MPI startup(): RDMA data transfer mode
--> Gigabit
[root@cn035 ~]# time -p /soft/intel/impi/3.2.1.009/bin64/mpiexec -np 16 -env I_MPI_DEBUG 2 /home/c/OMB-3.1.1/osu_mbw_mr
[0] MPI startup(): DAPL provider ofa-v2-ib0 specified in DAPL configuration file /etc/dat.conf
...
[0] MPI startup(): RDMA, shared memory, and socket data transfer modes
Since the default is the RDSSM device, your two runs are identical. If you'd like to run over GigE, you need to specify the "sock" device. Alternatively, you can use "ssm" for GigE communication between the nodes, and shared memory within a node. That means your command line would look like this:
# time -p /soft/intel/impi/3.2.1.009/bin64/mpiexec -np 16 -env I_MPI_DEBUG 2 -env I_MPI_DEVICE ssm /home/c/OMB-3.1.1/osu_mbw_mr
I hope this helps.
Regards,
~Gergana
--------
Gergana Slavova
Technical Consulting Engineer
Intel® Cluster Tools
E-mail: gergana.s.slavova_at_intel.com
