[openib-general] [ANNOUNCE] Contribute RDS (Reliable DatagramSockets) to OpenIB

[Ranjit Pandit]

On 11/4/05, Grant Grundler <iod00d at hp.com> wrote:
> On Thu, Nov 03, 2005 at 08:06:21PM -0500, Rick Frank wrote:
> > I've atttached a draft proposal for RDS from Oracle which discusses some of
> > the motivation for RDS.
>
> Thanks!
>
> Some questions/comments...
>
> o What is "GE" acronym for?

Gigabit Ethernet

>
> o I'm seeing about 1/5th CPU load for SDP (vs IPoIB).
>   The "50% less" number doesn't seem that impressive for RDP (vs IPoIB).
>   Maybe this is a difference in the benchmark (I'm running netperf).

1/5th CPU load with SDP is with zero copy or without?

>
> o RDP wants to provide AF_INET_OFFLOAD. This doesn't exist in my source tree.
>   I don't know who assigns these but it isn't lanana.org.
>   Oracle would be wise to stick with what's in include/linux/sockets.h
>   in order to avoid long term maintenance issues.
>
>   ISTR OpenIB got flamed for wanting to use AF_INET_OFFLOAD name.
>   If RDP is accepted, I would expect RDP to get AF_INET_RDP.
>   And then use "LD_PRELOAD" and clone libsdp.so to take over AF_INET.
>   ie follow a similar trajectory that SDP had.
>

On SST stack, AF_INET_OFFLOAD is used for both SDP as well as RDS. The
difference is in the socket_type, SOCK_STREAM Vs SOCK_DGRAM.

Can something similar be done on OpenIB?

> o Is access control to the RDP protocol something that applies to
>   all protocols?
>   I'm looking item #2 of "Additional Features".
>
In this particular case, Oracle had a specific requirement for access
control on RDP.
I don't know if other users will have similar requirement on other ULPs or not.

>
> o Doesn't SDP meet the following requirement as well?
>
> | A goal of RDP should be to support all existing socket
> | functionality relevant to UDP with no changes to any
> | existing socket application - other than specifying
> | AF_INET_OFFLOAD. However, an RDP aware socket application
> | can take advantage of the RDP features.
>

SDP does not support SOCK_DGRAM...

ulp/sdp/sdp_inet.c

 if (SOCK_STREAM != sock->type ||
            (IPPROTO_IP != protocol && IPPROTO_TCP != protocol)) {
                sdp_dbg_warn(NULL, "SOCKET: unsupported type/proto. <%d:%d>",
                             sock->type, protocol);

                return -EPROTONOSUPPORT;
        }

>
> o I'm struggling with the "RDP is connectionless" comments made earlier.
>   Later in this proposal, "RDP Interface" says packets will be
>   delivered "in order". Doesn't that conflict with "connectionless"?
>   Does UDP guarantee order?
>

As Fab and Rick mentioned, RDS provides UDP like connectionless model
to applications but it uses IB/RC to communicate to the remote node.
So the application doesn't have to maintain a connection state, which
is a problem with TCP/SDP when there are 100K odd connections
involved.

> o The "crossover" value for zero copy vs inlining data is chipset specific.
>   Ie even within the same architecture, different combinations of CPUs
>   and chipsets will give wide variance. Things like cache size, cache
>   replacement algorithm, available memory bandwidth, memory latency,
>   et al, affect the choice. This value is normally define by/for each
>   architecture since that's practical and lets each arch decide
>   what the right tradeoff is.
>
Agreed.

> o The comments in "Recv operations" talk about "backpressure".
>   Is this another way of saying the driver should drop packets once
>   the "fairness threshold" is exceeded?
>
The driver cannot drop packets.
When backpressure'd, RDS returns EWOULDBLOCK to the application and
then the application can retry.

> o Does detecting the "death of a remote node" still fall
>   within the "connectionless" definition?

When a particular socket is "backpressure'd/stalled" the application
gets EWOULDBLOCK. Meanwhile, if the destination node dies, that socket
needs to be unblocked. Any subsequent sends will return an error so
the application can take corrective measures or cleanup.

>
> o I didn't look through the "config" and "statistics".
>
> o "RDP Information" section reminds me of the previous email thread
>   about "netstat" support. Those probably want to be aligned so
>   Oracle can leverage the same command as other users.
>   ie reduce long term maintenance.
>
>
> And while researching the above, I found some nits with SDP:
>
> o I was expecting AF_INET_SDP to be in 2.6.14 and it's not.
>   I hope it's part of 2.6.15-rc*.
>
> o The ulp/sdp/Kconfig comments say "AF_INET_SDP (address family 26)".
>   AF_LLC uses 26 and sdp_sock.h defines 27.
>   Michael - need a patch or is this trivial enough to fix by hand?
>
> thanks,
> grant
> _______________________________________________
> openib-general mailing list
> openib-general at openib.org
> http://openib.org/mailman/listinfo/openib-general
>
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
>