[openib-general] ip over ib throughtput

Tue May 9 14:14:00 PDT 2006

Shirley, Hassan - I am *very* interested in these results, and I
want to at least mention that I'm doing similar NFS/RDMA testing,
and getting some contrasting results.

> 699040 699040  16384    60.00    3668.07 (458MB/s) 
> cpu utilization was around 95%. 

On my dual-2.4GHz Xeon, with the relatively untuned NFS/RDMA
client on 2.6.16.6, I am able to pull about 450MB/sec of read
throughput at 35% total CPU.

This is using 16 threads of NFS direct i/o (O_DIRECT) to a midrange
NetApp server, I did achieve a similar result with the Linux NFS/RDMA
server (but only after hotwiring the ext2 interface because I don't
have the spindles). I am using a dedicated filesystem test to generate
the load, and also iozone. These NFS/RDMA direct reads use RDMA
writes from the server to the client.

Also, this was with client hyperthreading disabled and a dual-processor
Dell, I could reboot with a single CPU to get more comparable results.
But, the throughput was limited by server CPU (100%), the client was
actually loafing a little bit.

I thought it was interesting that a filesystem achieves the same
throughput at better overhead than a dedicated network test. :-) 
And I haven't played with interrupt affinity at all.

Tom.

At 07:23 PM 5/8/2006, Shirley Ma wrote:

>I am testing most of my patches. Under 
>
>1.Intel(R) Xeon(TM) CPU 2.80GHz, one cpu, 
>2. fw-23108-3_4_000-MHXL-CF128-T.bin 
>3. pci-x without msi_x enabled 
>4. kernel 2.6.16 
>5. netperf-2.4.0 
>6. SVN 68XX+several IPoIB patches 
>
>The best result I got so far: 
>
>Testing with the following command line: 
>netperf -l 60 -H 10.1.1.100 -t TCP_STREAM -i 10,2 -I 95,5 -- -m 16384 -s 349520 -S 349520 
>
>TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.1.1.100 (10.1.1.100) port 0 AF_INET : +/-2.5% @ 95% conf. 
>Recv   Send    Send 
>Socket Socket  Message  Elapsed 
>Size   Size    Size     Time     Throughput 
>bytes  bytes   bytes    secs.    10^6bits/sec 
>
>699040 699040  16384    60.00    3668.07 (458MB/s) 
>
>cpu utilization was around 95%. 
>
>Thanks
>Shirley Ma
>IBM Linux Technology Center
>15300 SW Koll Parkway
>Beaverton, OR 97006-6063
>Phone(Fax): (503) 578-7638
>
>
>
>
>"Hassan M. Jafri" <hjafri at ncsa.uiuc.edu> 
>Sent by: openib-general-bounces at openib.org 
>
>05/08/2006 03:52 PM 
>To
>openib-general at openib.org 
>cc
>Subject
>Re: [openib-general] ip over ib throughtput 
>
>
>
>
>I cant crank out more than 150 MB/sec with my 2.0 GHz xeons. verbs level 
>benchmarks, however give decent numbers for bandwidth. With netperf, the 
>server side CPU usage is 99% which is much higher than other posted 
>bandwidth results on this thread. Any suggestions?
>
>Here is the complete configuration for my bandwidth tests
>
>Kernel-2.6.15.4
>netperf-2.3-3
>OpenIB rev 6552
>MTLP23108-CF128
>Firmware 3.4.0
>MSI-X is enabled for the HCA
>
>
>------------------------------
>Here is the netperf output
>
>
>
>TCP STREAM TEST to 192.168.2.2
>Recv   Send    Send                          Utilization       Service 
>Demand
>Socket Socket  Message  Elapsed              Send     Recv     Send    Recv
>Size   Size    Size     Time     Throughput  local    remote   local 
>remote
>bytes  bytes   bytes    secs.    MBytes  /s  % T      % T      us/KB   us/KB
>
>262142 262142  32768    10.01       151.32   59.66    99.84    7.700 
>12.886
>-------------------------------
>
>Here is ib0 config for one of the nodes
>
>ib0       Link encap:UNSPEC  HWaddr 
>00-02-04-04-FE-80-00-00-00-00-00-00-00-00-00-00
>          inet addr:192.168.2.1  Bcast:192.168.2.255  Mask:255.255.255.0
>          inet6 addr: fe80::202:c902:0:3ce9/64 Scope:Link
>          UP BROADCAST RUNNING MULTICAST  MTU:2044  Metric:1
>          RX packets:1724527 errors:0 dropped:0 overruns:0 frame:0
>          TX packets:9685456 errors:0 dropped:2 overruns:0 carrier:0
>          collisions:0 txqueuelen:128
>          RX bytes:89830114 (85.6 MiB)  TX bytes:2213308646 (2.0 GiB)
>
>
>
>
>
>
>
>
>Michael S. Tsirkin wrote:
>> Hi!
>> What kind of performance do people see with ip over ib on gen2?
>> I see about 100Mbyte/sec at 99% CPU utilisation on send,
>> on an express card, Xeon 2.8GHz, SSE doorbells enabled.
>> 
>> MST
>> _______________________________________________
>> openib-general mailing list
>> openib-general at openib.org
>> http://openib.org/mailman/listinfo/openib-general
>> 
>> To unsubscribe, please visit 
>> http://openib.org/mailman/listinfo/openib-general
>> 
>_______________________________________________
>openib-general mailing list
>openib-general at openib.org
>http://openib.org/mailman/listinfo/openib-general
>
>To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
>
>_______________________________________________
>openib-general mailing list
>openib-general at openib.org
>http://openib.org/mailman/listinfo/openib-general
>
>To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general