[openib-general] ib_rdma_bw measures 1.2G vs. 1.4G

[Bryan Green]

Hello,
I've been testing rdma bandwidth between a number
of machines using ib_rdma_bw, and I consistently
see two approximate bandwidths, 1.4 GBytes/s or
1.2 GBytes/s.
The 1.4G/s rate is what I expect from the link,
but I don't know why in some cases I get 1.2G/s.

What could cause this particular quantized
degradation in performance?  So far, these are
the datapoints I have (all systems are Mellanox DDR, 8x PCI-E):

There are 7 machines:
 ZeonA, ZeonB:  dual-Core2 zeon systems running Suse 10.1, OFED 1.0
 OptiA, OptiB, OptiC: older dual-cpu/dual-core Opteron systems
  running Gentoo with specialized 2.6.15 kernel, openib-1.0 userland.
 OptiX, OptiZ: brand new dual-cpu/dual-core Opteron systems,
  running Gentoo with 2.6.17-gentoo kernel, openib-1.1 userland.

Between ZeonA, ZeonB:
    1.4 G/s
Between OptiA, OptiB, and OptiC:
    1.2 G/s
Between OptiB and OptiC after kernel upgrades to 2.6.17-gentoo:
    1.4 G/s
Between OptiA (2.6.15) and OptiB (2.6.17-gentoo):
    1.4 G/s

Between OptiX, OptiZ:
    1.2 G/s

Between OptiX, ZeonA:
    1.2 G/s

I'm bummed to be seeing 1.2 G/s on the newer systems
with the 2.6.17 kernel.  What might be the explanation?

-bryan