[openib-general] ip over ib throughtput

[Grant Grundler]

On Wed, Jan 12, 2005 at 05:34:06PM -0800, Grant Grundler wrote:
> RX2600 sending TCP_Stream packets to RX4640 via topspin 12port switch.
> Rx2600 has "Low Profile" (Cougarcub) and rx4640 has "Cougar" installed
> in "dual rope" slots.
> 
> /opt/netperf/netperf -l 60 -H 10.0.1.81 -t TCP_STREAM -i 5,2 -I 99,5 -- -m 8192 -s 262144 -S 262144
...
> _spin_unlock_irqrestore is a clue we are spending time in
> interrupt handlers and that isn't getting measured.

David Mosberger gently reminded me he had added "-i" option
to monitor interrupt context as well.
Here is another run with "-i".

ionize:~# q-syscollect -i -k /opt/netperf/netperf -l 60 -H 10.0.1.81 -t TCP_STREAM -- -m 8192 -s 262144 -S 262144
TCP STREAM TEST to 10.0.1.81
Recv   Send    Send                          
Socket Socket  Message  Elapsed              
Size   Size    Size     Time     Throughput  
bytes  bytes   bytes    secs.    10^6bits/sec  

262142 262142   8192    60.00    1542.81   

q-view .q/kernel-cpu0.info#0 | less
Flat profile of CPU_CYCLES in kernel-cpu0.hist#0:
 Each histogram sample counts as 1.00034m seconds
% time      self     cumul     calls self/call  tot/call name
 27.91     16.69     16.69         -         -         - default_idle
  5.48      3.27     19.96         -         -         - ipt_do_table
  4.28      2.56     22.52         -         -         - do_csum
  3.81      2.28     24.80         -         -         - __copy_user
  2.53      1.51     26.32         -         -         - local_bh_enable
  2.52      1.51     27.82         -         -         - tcp_transmit_skb
  2.51      1.50     29.33         -         -         - nf_iterate
  2.33      1.39     30.72         -         -         - mthca_poll_cq
  2.19      1.31     32.03         -         -         - tcp_sendmsg
  1.73      1.03     33.06         -         -         - mthca_post_send
  1.48      0.88     33.95         -         -         - ip_queue_xmit
  1.33      0.80     34.74         -         -         - ip_conntrack_in
  1.31      0.79     35.53         -         -         - mthca_eq_int
  1.23      0.73     36.26         -         -         - _spin_lock
...


cpu1 was 99.87% of the time in default_idle.

hth,
grant