[openib-general] [ANNOUNCE]ContributeRDS(ReliableDatagramSockets) to OpenIB

[Rimmer, Todd]

> Fab wrote,
> >There is not a 1:1 relationship between a UDP application 
> socket and an IB
> QP,
> >rather there is a single IB connection between systems over 
> which traffic
> from
> >multiple UDP sockets flows.
> 
> >- Fab
>
> Bob wrote,
> That would probably provide better scalability, since there
> would not be a 1:1 mapping between UDP sockets and IB connections,
> however for large clusters there may still be a scalability issue
> if every node needs to have a connection to every other node. 
> If you implemented it on top of datagrams instead, then each node
> would only need one QP, rather than one for every node in the cluster.

That is essentially what Oracle previously did when using UDP over IPoIB.

Significant performance gains were realized with RDS (as compared to IPoIB) for a number of reasons:
1. use of RC connections allows for messages larger than IB MTU, which allows for more efficiency and better performance.
2. By using RC connections and flow control in the RDS socket mux, Oracle was able to remove the need for timeouts and retries in application space.  Such algorithms in application space can get expensive, especially due to error handling which is inevitable when congestion and stress force the loss of packets by any unreliable datagram protocol (IB/UD or IPoIB/UDP or Ethernet/UDP).

Todd Rimmer