[openib-general] Re: user-mode verbs on Itanium

[Grant Grundler]

On Fri, Apr 29, 2005 at 11:38:46AM -0700, Grant Grundler wrote:
...
> ionize:~# ibv_pingpong -s 64 -n 100000 10.0.0.51
>   local address:  LID 0x000d, QPN 0x070406, PSN 0x972e9e
>   remote address: LID 0x000b, QPN 0x030406, PSN 0x3c7543
> 12800000 bytes in 1.65 seconds = 62.10 Mbit/sec
> 100000 iters in 1.65 seconds = 16.49 usec/iter
...
> ionize:~# ibv_pingpong -s 65536 -n 10000
>   local address:  LID 0x000d, QPN 0x0b0406, PSN 0xd79a37
>   remote address: LID 0x000b, QPN 0x070406, PSN 0x940ac7
> 1310720000 bytes in 2.84 seconds = 3691.26 Mbit/sec
> 10000 iters in 2.84 seconds = 284.07 usec/iter

It seemed the numbers were too low and of course "ionize" had the IB card
in a "1/2 bandwidth" slot (aka "single rope").
3691 Mbit/sec is about 90% utilization (461:512 MB/s) of the rope.

Re-running the extremes, I'm getting better numbers: 
grundler at gsyprf3:~$ ibv_pingpong -s 65536 -n 100000 10.0.0.30
  local address:  LID 0x000b, QPN 0x180406, PSN 0x7609fa
  remote address: LID 0x0010, QPN 0x010406, PSN 0xf10582
13107200000 bytes in 17.99 seconds = 5830.14 Mbit/sec
100000 iters in 17.99 seconds = 179.85 usec/iter

5830 Mbit/s still seems low for this box...only 728 MB/s.
I'm expecting at least 800 MB/s and about 900 MB/s
per link under "reasonable" conditions (ie > 1K size msgs).
Is the issue ibv_pingpong is a single threaded test?
Should I run a script to start/send several instances
of ibv_pingpong on different ports?


grundler at gsyprf3:~$ ibv_pingpong -s 64 -n 1000000 10.0.0.30
  local address:  LID 0x000b, QPN 0x190406, PSN 0x04c55b
  remote address: LID 0x0010, QPN 0x020406, PSN 0x350ff5
128000000 bytes in 15.96 seconds = 64.17 Mbit/sec
1000000 iters in 15.96 seconds = 15.96 usec/iter

Is the 15.96 usec/iter the round trip time?

thanks,
grant