[ofa-general] SDP and iWARP

[Craig Prescott]

Steve Wise wrote:
> Roland Dreier wrote:
>> Sorry to come into this thread so late, but does it make sense to try
>> the current SDP code over iWARP?  As I understand things, the RDMA
>> consortium has its own spec for SDP on iWARP, which may not precisely
>> correspond to the IBA SDP annex.  So probably the SDP code would need
>> updating to work over iWARP.
>>
>
> I didn't think they were that different, but I don't know for sure. 
> However, unless the IB-SDP uses atomics or some other IB-specific work 
> request, it just might work.
>
Sorry for the slow follow-up.  SDP on iWARP is working now:

[root at tebow2 ~]# /opt/netperf/bin/netperf -H 128.227.253.91 -L 
128.227.253.92 -t SDP_STREAM -c -C -l 10 -p 5006
SDP STREAM TEST from 128.227.253.92 (128.227.253.92) port 0 AF_INET to 
128.227.253.91 (128.227.253.91) port 0 AF_INET
Recv   Send    Send                          Utilization       Service 
Demand
Socket Socket  Message  Elapsed              Send     Recv     Send    Recv
Size   Size    Size     Time     Throughput  local    remote   local   
remote
bytes  bytes   bytes    secs.    10^6bits/s  % S      % S      us/KB   us/KB

262144 262144 262144    10.00      6305.54   16.39    14.38    0.852   
1.495 

The patch to enable this is not big - I will produce one and send it to
the list.  Might not happen before next week.

There is only one other remarkable problem encountered which is not
already documented in this thread.  That is, when SDP tries to resize
the receive private buffers (receiver gets an SDP_MID_CHRCVBUF), this
can create up to 9 scatter-gather entries for each associated work
request.  This is larger than the Chelsio RNICs I am using can handle
(T3_MAX_SGE), and the building of work requests fails.

Does T3_MAX_SGE come from hardware?

Anyway, one way to work around this was to deny SDP any "large" sockets
via moddule parameter (max_large_sockets=0).  It would be good if SDP
queried the RNIC for the max number of SGEs when an SDP_MID_CHRCVBUF
is encountered, and resize the private buffers in a way that will not
exceed the capability of the device.

I haven't tried that yet, but I have limited the requested
receive buffer size to something the Chelsio RNIC could handle.
The above netperf result uses this hack.

Thanks again for all your help.  Will post some numbers shortly.

Cheers,
Craig