<html>
<body>
<font size=3>At 08:05 AM 5/27/2005, Bernard Metzler wrote:<br><br>
</font><blockquote type=cite class=cite cite=""><font size=2>
Sukanta,</font><font size=3> <br><br>
</font><font size=2>without touching any TOE issues (this question is
about RDMA, right?), after</font><font size=3> <br>
</font><font size=2>transforming a TCP connection into RDMA mode and
using an RDMA API,</font><font size=3> <br>
</font><font size=2>the socket file descriptor is not longer to be used
for communication.</font><font size=3> </font></blockquote><br>
Whether one uses a Socket FD or not is an application issue. A QP
has a handle identifier which may be mapped to a FD or may be used
directly by an application. There is no requirement that anything
flow through the Sockets API - that is a choice for the application to
make. <br><br>
<blockquote type=cite class=cite cite=""><font size=2>In fact, on some
implementations the stream socket resources will</font><font size=3>
</font><font size=2>even get released. That is, if the socket was the
direct consumer</font><font size=3> </font><font size=2>of the TCP
stream, then now it is RDMAP/DDP/MPA.</font><font size=3>
</font><font size=2>RDMA APIs such as IT-API defining a specific call to
convert a socket based</font><font size=3> </font><font size=2>connection
into RDMA mode (e.g., it_socket_convert()).</font><font size=3>
</font><font size=2>Other consumers may directly start via an RDMA API,
never</font><font size=3> <br>
</font><font size=2>opening a consumer controlled
socket.</font><font size=3> </font></blockquote><br>
Correct.<br><br>
<blockquote type=cite class=cite cite=""><font size=2>So, in RDMA mode,
communication will happen via the RDMA API. At this</font><font size=3>
</font><font size=2>stage, the kernel still have to keep completely in
its hands the</font><font size=3> </font><font size=2>synchronisation of
state information related to that offloaded</font><font size=3>
</font><font size=2>connection(s) with the host stack (it would have to
protect the</font><font size=3> </font><font size=2>local port used by
the offloaded connection for example, others</font><font size=3>
</font><font size=2>are routing, ARP, SNMP...), but it is not involved at
the data path.</font><font size=3> </font></blockquote><br>
This is true only when the host and the RNIC share a common IP
address. If they are separate subnets, then there is no need to
coordinate sans some of the reasons I noted in an earlier
e-mail. I think there is great value and customer need to
have an infrastructure that can coordinate information and enable
customers to use the existing tool chains to understand what is going on
in a given device or endnode.<br><br>
<blockquote type=cite class=cite cite=""><font size=2>With respect to the
kernel based TCP stack, what is not needed is</font><font size=3>
</font><font size=2>a hack into the stack and scatter/gather state
information of the live</font><font size=3> </font><font size=2>TCP
connection between kernel and RNIC, but to find one clean
interface</font><font size=3> </font><font size=2>to transfer state
information out of that stack and to the RNIC.</font><font size=3>
</font></blockquote><br>
At a minimum, one needs:<br><br>
- A host network stack get state call to extract connection state and
quiesce the port from the host's perspective.<br>
- A host network stack set state call to populate the host structures
with the associated state and enable the port<br>
- A RNIC get state call to extract all transport and RMDA context<br>
- A RNIC set state call to populate all transport and RDMA
context<br><br>
These calls should be "standardized" so that both sides of the
infrastructure will be able to utilize without having to hack
anything.<br><br>
<blockquote type=cite class=cite cite=""><font size=2>With limited
benefit, one could of course also implement native sockets over RDMA,
where an in-kernel midlayer on top of kernel</font><font size=3> <br>
</font><font size=2>RDMA Verbs is doing the translation between send(),
receive() to post_send, post_receive. But usage of 'true RDMA'
operations</font><font size=3> <br>
</font><font size=2>like RDMA READ or WRITE might be limited, and I don't
see much value for the user here. One variety of this approach with less
limited</font><font size=3> </font><font size=2>access to RMDA benefits
might be sockets with extended RDMA semantics.</font><font size=3>
</font></blockquote><br>
Sockets with RDMA extensions was proposed a couple of years back by some
of us. There would be quite a bit of benefit as most developers
know how to code to Sockets and the RDMA extensions are relatively simple
when you think about it. It would also eliminate the need to use
SDP by placing the RDMA knowledge requirement on the application.
SDP has benefits for enabling existing Sockets applications to operate
transparently over RDMA but the explicit use of RDMA extensions I think
would provide greater benefit in the end. While it is important to
provide legacy investment protection, real innovation will not occur
until people start developing technology that enables a smarter
customer. Simplification and performance also come with enabling a
smarter customer. <br><br>
Mike<br><br>
<br>
<blockquote type=cite class=cite cite=""><font size=2>Bernard.</font>
<font size=3> <br><br>
</font><tt>rdma-developers-admin@lists.sourceforge.net wrote on
27.05.2005 15:40:43:<br><br>
> Venkata,<br>
> How will that work? If the RNIC offloads RDMA
and<br>
> TCP completely from the Operating System and does not<br>
> share any state information then the application<br>
> running on the host will never be in the position to<br>
> utilize the socket interface to use the communication<br>
> logic to send and receive data between the remote node<br>
> and itself. Some information needs to be shared. How<br>
> much of it and what exactly needs to be shared is the<br>
> question.<br>
> <br>
> Thanks<br>
> SG<br>
> <br>
> --- Venkata Jagana <jagana@us.ibm.com> wrote:<br>
> > <br>
> > <br>
> > <br>
> > <br>
> > <br>
> > <br>
> > rdma-developers-admin@lists.sourceforge.net wrote on<br>
> > 05/25/2005 09:47:00<br>
> > PM:<br>
> > <br>
> > > Venkata,<br>
> > > Interesting coincidence: I was talking with<br>
> > someone (at HP) today<br>
> > > who knows substantially more than I do about<br>
> > RNICs.<br>
> > > They indicated RNICs need to manage TCP state on<br>
> > the card from userspace.<br>
> > > I suspect that's only possible through a private<br>
> > interface<br>
> > > (e.g. ioctl() or /proc) or the non-existant (in<br>
> > kernel.org)<br>
> > > TOE implementation. Is this correct?<br>
> > ><br>
> > <br>
> > Not correct.<br>
> > <br>
> > Since RNICs are offloaded adapters with RDMA<br>
> > protocols layered on<br>
> > top of TCP stack, they do maintain the TCP state<br>
> > internally but<br>
> > it does not expose to the host. RNIC expose only<br>
> > RNIC Verbs interface<br>
> > to the host bot not TOE interface.<br>
> > <br>
> > Thanks<br>
> > Venkat<br>
> > <br>
> > ><br>
> > > hth,<br>
> > > grant<br>
> > ><br>
> > ><br>
> > ><br>
> ><br>
> -------------------------------------------------------<br>
> > > SF.Net email is sponsored by: GoToMeeting - the<br>
> > easiest way to<br>
> > collaborate<br>
> > > online with coworkers and clients while avoiding<br>
> > the high cost of travel<br>
> > and<br>
> > > communications. There is no equipment to buy and<br>
> > you can meet as often as<br>
> > > you want. Try it<br>
> ><br>
>
free.http://ads.osdn.com/?ad_id=7402&alloc_id=16135&op=click<br>
> > > _______________________________________________<br>
> > > Rdma-developers mailing list<br>
> > > Rdma-developers@lists.sourceforge.net<br>
> > >
<a href="https://lists.sourceforge.net/lists/listinfo/rdma-developers" eudora="autourl">
https://lists.sourceforge.net/lists/listinfo/rdma-developers</a><br>
> <br>
> __________________________________________________<br>
> Do You Yahoo!?<br>
> Tired of spam? Yahoo! Mail has the best spam protection around
<br>
>
<a href="http://mail.yahoo.com/" eudora="autourl">
http://mail.yahoo.com</a> <br>
> <br>
> <br>
> -------------------------------------------------------<br>
> This SF.Net email is sponsored by Yahoo.<br>
> Introducing Yahoo! Search Developer Network - Create apps using
Yahoo!<br>
> Search APIs Find out how you can build Yahoo! directly into your
own<br>
> Applications - visit
<a href="http://developer.yahoo.net/?fr=offad-ysdn-ostg-q22005" eudora="autourl">
http://developer.yahoo.net/?fr=offad-ysdn-ostg-q22005</a><br>
> _______________________________________________<br>
> Rdma-developers mailing list<br>
> Rdma-developers@lists.sourceforge.net<br>
>
<a href="https://lists.sourceforge.net/lists/listinfo/rdma-developers" eudora="autourl">
https://lists.sourceforge.net/lists/listinfo/rdma-developers</a><br>
</tt><font size=3>_______________________________________________<br>
openib-general mailing list<br>
openib-general@openib.org<br>
<a href="http://openib.org/mailman/listinfo/openib-general" eudora="autourl">
http://openib.org/mailman/listinfo/openib-general</a><br><br>
To unsubscribe, please visit
<a href="http://openib.org/mailman/listinfo/openib-general" eudora="autourl">
http://openib.org/mailman/listinfo/openib-general</a>
</font></blockquote></body>
</html>