[openib-general] Re: Re: user-mode verbs on Itanium

[Michael S. Tsirkin]

Quoting r. Grant Grundler <iod00d at hp.com>:
> Subject: Re: Re: user-mode verbs on Itanium
> 
> On Fri, Apr 29, 2005 at 11:38:46AM -0700, Grant Grundler wrote:
> ...
> > ionize:~# ibv_pingpong -s 64 -n 100000 10.0.0.51
> >   local address:  LID 0x000d, QPN 0x070406, PSN 0x972e9e
> >   remote address: LID 0x000b, QPN 0x030406, PSN 0x3c7543
> > 12800000 bytes in 1.65 seconds = 62.10 Mbit/sec
> > 100000 iters in 1.65 seconds = 16.49 usec/iter
> ...
> > ionize:~# ibv_pingpong -s 65536 -n 10000
> >   local address:  LID 0x000d, QPN 0x0b0406, PSN 0xd79a37
> >   remote address: LID 0x000b, QPN 0x070406, PSN 0x940ac7
> > 1310720000 bytes in 2.84 seconds = 3691.26 Mbit/sec
> > 10000 iters in 2.84 seconds = 284.07 usec/iter
> 
> It seemed the numbers were too low and of course "ionize" had the IB card
> in a "1/2 bandwidth" slot (aka "single rope").
> 3691 Mbit/sec is about 90% utilization (461:512 MB/s) of the rope.
> 
> Re-running the extremes, I'm getting better numbers: 
> grundler at gsyprf3:~$ ibv_pingpong -s 65536 -n 100000 10.0.0.30
>   local address:  LID 0x000b, QPN 0x180406, PSN 0x7609fa
>   remote address: LID 0x0010, QPN 0x010406, PSN 0xf10582
> 13107200000 bytes in 17.99 seconds = 5830.14 Mbit/sec
> 100000 iters in 17.99 seconds = 179.85 usec/iter
> 
> 5830 Mbit/s still seems low for this box...only 728 MB/s.
> I'm expecting at least 800 MB/s and about 900 MB/s
> per link under "reasonable" conditions (ie > 1K size msgs).
> Is the issue ibv_pingpong is a single threaded test?
> Should I run a script to start/send several instances
> of ibv_pingpong on different ports?
> 
> 
> grundler at gsyprf3:~$ ibv_pingpong -s 64 -n 1000000 10.0.0.30
>   local address:  LID 0x000b, QPN 0x190406, PSN 0x04c55b
>   remote address: LID 0x0010, QPN 0x020406, PSN 0x350ff5
> 128000000 bytes in 15.96 seconds = 64.17 Mbit/sec
> 1000000 iters in 15.96 seconds = 15.96 usec/iter
> 
> Is the 15.96 usec/iter the round trip time?
> 
> thanks,
> grant

I think ibv_pingpong is not the best latency benchmark - more of
a test/usage example. Since it has no pipelining, its also not
the best bandwidth benchmark.

Grant, if you want a latency benchmark you may want to look at
contrib/mellanox/perftest.
Its a variation on the pingpong test, showing how you can get latency
between 3 and 4usec one-way, by using inline data on sends, and polling on data.
I expect to put up a bandwidth test one of these days.

-- 
MST - Michael S. Tsirkin