[ofa-general] Poor Performance of OpenIB with small packets c.f. Gigabit Ethernet

[David Robb]

We have a data logging application that exhibits poor performance when 
operated using TCP/IP sockets and IPoIB.

With small message sizes ~ 64 bytes, the performance values for our 
application are

OFED 1.2 IPoIB:     2.81MB/s
OFED 1.3 IPoIB:     1.37MB/s
GB Ethernet:            5.38MB/s

It is not until the message sizes reach 16K or so that the Infiniband 
starts to overtake the Ethernet.

Are these values as expected?

What further tests could I run to investigate the problem?

Are there any settings and or device configuration that we can tweak to 
improve the small message performance?

We are running RH-EL Linux and using Mellanox HCAs and switches.
We recently upgrade to OFED 1.3 and have upgraded the HCA firmware to 
the latest 1.2 version.

Many thanks for any help

Regards

David Robb