[ofa-general] 1MB/sec IPoIB performance IBM Blade and RHEL 5.3?

[Craig Reese]

Not sure if this is a RedHat, IBM, or OpenFabrics issue, but thought 
this might be a good place to start....

Environment:

IBM HS20 (8843), each with 1x IB card (26K6456: board_id: HCA.Cougar.A1)
IBM/TopSpin switch module (26K6453)
RHEL 5.3 Server w/ OFED 1.3.1 (as comes with the 5.3 distribution)
(I know this is somewhat dated, but it's what I have to experiment with 
at the moment)

I pretty much just installed things and started them up. No tuning  or 
updating of software etc....

I've got interfaces setup on each machine: 

  hs20[ab] = Ethernet
  hs20[ab]-ib = Infiniband

both seem to be working for normal IP stuff (ssh, ping, etc...)

I was curious about performance so I ran qperf:

starting the server on hs20a with : 

  hs20a> qperf

running the client from hs20b:

  hs20b> qperf hs20a tcp_bw
  tcp_bw:
      bw = 118 MB/sec
  hs20b> qperf hs20a-ib tcp_bw
  tcp_bw:
      bw = 1.15 MB/sec                       <-- Not a typo!
  hs20b> qperf hs20a-ib sdp_bw
  sdp_bw:
      bw = 238 MB/sec

I got similar results using netperf.

Can anyone account for the poor performance in case 2?  I'm unclear
whether it's obsolete hardware, misconfigured software/hardware, 
deficiencies in
1.3.1 software, or my misinterpretation of the results, but 1.15MB/sec seems
pretty bad.