[openib-general] RDMA Generic Connection Management

[Talpey, Thomas]

At 10:55 AM 8/30/2005, Yaron Haviv wrote:
>The iSCSI discovery may return multiple src & dst IP addresses and the
>iSCSI multipath implementation will open multiple connections.
>There are many TCP/IP protocols that do that at the upper layers (e.g.
>GridFTP, ..), not sure how NFS does it.

The answer to that question depends on the version of NFS, and
also the implementation.

For NFSv2/v3, the situation is ad hoc. Some clients support
multiple connections which they are able to round-robin. Solaris
does this for example. The problem is, to the server each NFSv2/v3
connection appears to be a different client. Therefore the
correctness guarantees (such as they are) go out the window.
For example, a retry on a different connection is not a retry at
all, it's a new op. So, the shotgun (trunked) NFSv3 situation is
useful only for a certain class of use.

For NFSv4, it's a little better in that there is a clientid which
identifies the source. However, NFSv4 does not sufficiently deal
with the case of requests on different connections either. 

With our new NFSv4 sessions proposal, planned to be part of
NFSv4.1 (http://www.ietf.org/internet-drafts/draft-ietf-nfsv4-sess-02.txt),
trunking is fully supported, by allowing requests to belong to a
higher-layer session regardless of what connection they arrive
on. This exists in prototype form, the NFSv4.1 spec is still being
pulled together. UMich/CITI is developing this btw. With a session,
the client gets full consistency guarantees and trunked connections
are therefore completely transparent.

One thing to stress is that the type of connection (TCP, UDP,
RDMA, etc) makes little or no difference in the trunking/multipathing
picture. In fact, with an NFSv4.1 session, a mix of such connections
is possible, and even a good idea. So it's more than a question of
what RDMA capabilities are there, it's really *all* connections.

To answer the question of how NFS "finds out" about multiple
connections and trunking, the answer is generally that the mount
command tells it. Mount can get this information from the command
line, or DNS. I believe Solaris uses the command line approach. There
may be a way to use the RPC portmapper for it, but the portmapper
isn't used by NFSv4.

Bottom line? NFS would love to have a way to learn multipathing
topology. But it needs to follow existing practice, such as having
an IP address / DNS expression. If the only way to find it is to query
fabric services, that's not very compelling.

Tom.