Hi everyone, I still evaluate cluster performance. For now, i move on virtualization with PCI-passthrough via FDR infiniband on KVM hypervisor.
My problem is Sendrecv throughput that decrease by half when compare with physical machine and i use 1 rank/node. For example
Node Bare-metal (MB/s) PCI-passthrough (MB/s)
2 14,600 13,000
4 14,500 12,000
16 14,300 11,000
32 14,290 10,000
64 14,200 7,100
What do you think about this behavior ? Is it about mellanox software or other things or overhead ?