[ofa-general] SDP and iWARP

[Steve Wise]

Craig Prescott wrote:
> Steve Wise wrote:
>> Roland Dreier wrote:
>>> Sorry to come into this thread so late, but does it make sense to try
>>> the current SDP code over iWARP?  As I understand things, the RDMA
>>> consortium has its own spec for SDP on iWARP, which may not precisely
>>> correspond to the IBA SDP annex.  So probably the SDP code would need
>>> updating to work over iWARP.
>>>
>>
>> I didn't think they were that different, but I don't know for sure. 
>> However, unless the IB-SDP uses atomics or some other IB-specific work 
>> request, it just might work.
>>
> Sorry for the slow follow-up.  SDP on iWARP is working now:
> 

Good work!

> [root at tebow2 ~]# /opt/netperf/bin/netperf -H 128.227.253.91 -L 
> 128.227.253.92 -t SDP_STREAM -c -C -l 10 -p 5006
> SDP STREAM TEST from 128.227.253.92 (128.227.253.92) port 0 AF_INET to 
> 128.227.253.91 (128.227.253.91) port 0 AF_INET
> Recv   Send    Send                          Utilization       Service 
> Demand
> Socket Socket  Message  Elapsed              Send     Recv     Send    Recv
> Size   Size    Size     Time     Throughput  local    remote   local   
> remote
> bytes  bytes   bytes    secs.    10^6bits/s  % S      % S      us/KB   
> us/KB
> 
> 262144 262144 262144    10.00      6305.54   16.39    14.38    0.852   
> 1.495
> The patch to enable this is not big - I will produce one and send it to
> the list.  Might not happen before next week.
> 

What mtu are you using?

> There is only one other remarkable problem encountered which is not
> already documented in this thread.  That is, when SDP tries to resize
> the receive private buffers (receiver gets an SDP_MID_CHRCVBUF), this
> can create up to 9 scatter-gather entries for each associated work
> request.  This is larger than the Chelsio RNICs I am using can handle
> (T3_MAX_SGE), and the building of work requests fails.
> 
> Does T3_MAX_SGE come from hardware?
> 

yes.

> Anyway, one way to work around this was to deny SDP any "large" sockets
> via moddule parameter (max_large_sockets=0).  It would be good if SDP
> queried the RNIC for the max number of SGEs when an SDP_MID_CHRCVBUF
> is encountered, and resize the private buffers in a way that will not
> exceed the capability of the device.
> 
> I haven't tried that yet, but I have limited the requested
> receive buffer size to something the Chelsio RNIC could handle.
> The above netperf result uses this hack.
> 
> Thanks again for all your help.  Will post some numbers shortly.
> 
> Cheers,
> Craig