Josh> I'm seeing about 364 MB/s between 2 PCIe Xeon 3.2GHz boxes
Josh> using netperf-2.3pl1.
Are you using MSI-X? To use it, set CONFIG_PCI_MSI=y when you build
your kernel and either "modprobe ib_mthca msi_x=1" or add "options
ib_mthca msi_x=1" to a file in /etc/modprobe.d. If MSI-X is enabled,
your /proc/interrupts will have mthca lines like
217: 0 0 26860275 0 PCI-MSI-X ib_mthca (comp)
225: 0 0 10 0 PCI-MSI-X ib_mthca (async)
233: 0 0 11572 0 PCI-MSI-X ib_mthca (cmd)
instead of
193: 3954254 2655769514 1116770165 1544790344 IO-APIC-level ib_mthca
If you're not using MSI-X you may be able to crack 400 MB/sec by turning it on.
- Roland