[ofa-general] Re: why is CPU util/service demand so much higher withSDP than TCP?

[Rick Jones]

Scott Weitzenkamp (sweitzen) wrote:
> # modinfo ib_sdp
> filename:
> /lib/modules/2.6.16.21-0.8-smp/updates/kernel/drivers/infiniband
> /ulp/sdp/ib_sdp.ko
> author:         Michael S. Tsirkin
> description:    InfiniBand SDP module
> license:        Dual BSD/GPL
> vermagic:       2.6.16.21-0.8-smp SMP gcc-4.1
> depends:        ib_core,rdma_cm
> srcversion:     91793E4825DEBC7A2DA9366
> parm:           top_mem_usage:Top system wide sdp memory usage for recv
> (in MB).
>  (int)
> parm:           rcvbuf_scale:Receive buffer size scale factor. (int)
> parm:           send_poll_thresh:Send message size thresh hold over
> which to sta
> rt polling. (int)
> parm:           recv_poll:How many times to poll recv. (int)
> parm:           send_poll:How many times to poll send. (int)
> parm:           recv_poll_miss:How many times recv poll missed. (int)
> parm:           recv_poll_hit:How many times recv poll helped. (int)
> parm:           send_poll_miss:How many times send poll missed. (int)
> parm:           send_poll_hit:How many times send poll helped. (int)
> parm:           data_debug_level:Enable data path debug tracing if > 0.
> (int)
> parm:           debug_level:Enable debug tracing if > 0. (int)
> 

I've learned a new command today :)  And via other channels how to actually see 
the current values of those things under /sys/modules/ib_sdp/parameters.

I think I may have also figured-out why my TCP_RR tests over IPoIB have been 
consistently unable to hit confidence intervals for CPU utilization.  I am 
guessing that for IPoIB that path is effectively "recv_poll=0 and/or 
send_poll=0" yes?  When I set those values for ib_sdp and run the SDP_RR test 
netperf is unable to be confident that the CPU util it measures is any closer 
than +/- 20% of the "real" CPU utilization...  (still the OFED bits in RHEL5 
rather than 1.2)  With the defaults, the netperf SDP_RR test hit the confidence 
intervals.

rick jones