[ofa-general] ***SPAM*** UD latency higher than RC on AMD quad core blades with ConnectX?

[Drake Samson]

Hello IB people,

I set up an AMD dual quad-core system recently and ran some IB level tests.
The "ibv_ud_pingpong" and "ibv_rc_pingpong" tests show pretty different
results for UD vs RC (latency is more than double!). I'm wondering if
someone could shed light on the issue? Is there something that needs to be
updated or changed? Has someone else noticed this phenomena?

$ numactl --physcpubind=0 --membind=0 ibv_ud_pingpong -s 1024 -d mlx4_0
  local address:  LID 0x003e, QPN 0x2c004a, PSN 0x7426cf
  remote address: LID 0x0045, QPN 0x2e004a, PSN 0x352c7e
2048000 bytes in 0.03 seconds = 609.23 Mbit/sec
1000 iters in 0.03 seconds = 26.89 usec/iter

$ numactl --physcpubind=0 --membind=0 ibv_rc_pingpong -s 1024 -d mlx4_0
  local address:  LID 0x003e, QPN 0x2e004a, PSN 0xf8fcb5
  remote address: LID 0x0045, QPN 0x30004a, PSN 0x221e94
2048000 bytes in 0.01 seconds = 1413.39 Mbit/sec
1000 iters in 0.01 seconds = 11.59 usec/iter

[there is no difference with/without numactl]

Here is the system description:

OS: Red Hat Enterprise Linux Server release 5.2 (Tikanga); kernel
2.6.18-92.el5
Processor: Quad-Core AMD Opteron(tm) Processor 2356
IB software: OFED-1.4
Firmware version: 2.5
Harware version: 0xA0
Vendor part id: 25418

-D
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20090406/885106b3/attachment.html>