[openib-general] Re: user-mode verbs on Itanium

[Grant Grundler]

On Fri, Apr 29, 2005 at 09:36:43AM -0700, Roland Dreier wrote:
>     Grant> But it's still not working: yprf3:~#
>     Grant> OPENIB_DRIVER_PATH=/usr/lib/infiniband/ ibv_pingpong
>     Grant> Couldn't get context for mthca0
> 
> Does the kernel driver match the userspace code -- ie are they both
> the latest subversion?

The only diff between r2225 and r2229 was this:
grundler at gsyprf3:/usr/src/linux-2.6/drivers/infiniband-r2229$ svn up -r 2229
U  core/cm.c
Updated to revision 2229.

That was it...I now get:
gsyprf3:/usr/src/linux-2.6# ibv_pingpong 
local address:  LID 0x000b, QPN 0x000406, PSN 0x5e48e6


And the other box, ionize, was ok (both using r2229). My bad.
I reloaded the kernel modules in ionize again just to be sure.

And WOOT! It Works. :^)

Reminder, I'm using:
	kernel.org 2.6.11, openib.org r2229, gcc 3.3.5 (Debian 1:3.3.5-12),
	rx2600 (ZX1 chipset), 1.5Ghz Madisons,
	MT23108 (PCI-X, Cougar) in dual rope slot.

ionize:/usr/src/openib_gen2/src/userspace# ibv_pingpong 10.0.0.51
  local address:  LID 0x000d, QPN 0x040406, PSN 0xfe84d1
  remote address: LID 0x000b, QPN 0x000406, PSN 0x5e48e6
8192000 bytes in 0.04 seconds = 1588.06 Mbit/sec
1000 iters in 0.04 seconds = 41.27 usec/iter

gsyprf3:/usr/src/linux-2.6# ibv_pingpong 
  local address:  LID 0x000b, QPN 0x000406, PSN 0x5e48e6
  remote address: LID 0x000d, QPN 0x040406, PSN 0xfe84d1
8192000 bytes in 0.04 seconds = 1592.19 Mbit/sec
1000 iters in 0.04 seconds = 41.16 usec/iter


And again with more iterations:
ionize:/usr/src/openib_gen2/src/userspace# ibv_pingpong -n 100000 10.0.0.51
  local address:  LID 0x000d, QPN 0x050406, PSN 0x5bf9ea
  remote address: LID 0x000b, QPN 0x010406, PSN 0x1e8764
819200000 bytes in 4.09 seconds = 1600.59 Mbit/sec
100000 iters in 4.09 seconds = 40.94 usec/iter


And a few more just for fun:

ionize:~# ibv_pingpong -s 128 -n 100000 10.0.0.51
  local address:  LID 0x000d, QPN 0x060406, PSN 0xbed9c7
  remote address: LID 0x000b, QPN 0x020406, PSN 0xf4b9db
25600000 bytes in 1.69 seconds = 120.95 Mbit/sec
100000 iters in 1.69 seconds = 16.93 usec/iter
ionize:~# ibv_pingpong -s 64 -n 100000 10.0.0.51
  local address:  LID 0x000d, QPN 0x070406, PSN 0x972e9e
  remote address: LID 0x000b, QPN 0x030406, PSN 0x3c7543
12800000 bytes in 1.65 seconds = 62.10 Mbit/sec
100000 iters in 1.65 seconds = 16.49 usec/iter

ionize:~# ibv_pingpong -s 16384 -n 10000 10.0.0.51
  local address:  LID 0x000d, QPN 0x080406, PSN 0xbe0533
  remote address: LID 0x000b, QPN 0x040406, PSN 0xca01ea
327680000 bytes in 1.16 seconds = 2267.26 Mbit/sec
10000 iters in 1.16 seconds = 115.62 usec/iter
ionize:~# ibv_pingpong -s 32768 -n 10000
  local address:  LID 0x000d, QPN 0x0a0406, PSN 0x7fc006
  remote address: LID 0x000b, QPN 0x060406, PSN 0xfdb40d
655360000 bytes in 1.51 seconds = 3471.72 Mbit/sec
10000 iters in 1.51 seconds = 151.02 usec/iter
ionize:~# ibv_pingpong -s 65536 -n 10000
  local address:  LID 0x000d, QPN 0x0b0406, PSN 0xd79a37
  remote address: LID 0x000b, QPN 0x070406, PSN 0x940ac7
1310720000 bytes in 2.84 seconds = 3691.26 Mbit/sec
10000 iters in 2.84 seconds = 284.07 usec/iter

thanks,
grant