<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META http-equiv=Content-Type content="text/html; charset=us-ascii">
<META content="MSHTML 6.00.2900.2722" name=GENERATOR></HEAD>
<BODY>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff
size=2></FONT> </DIV><BR>
<BLOCKQUOTE dir=ltr
style="PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT: #0000ff 2px solid; MARGIN-RIGHT: 0px">
<DIV class=OutlookMessageHeader lang=en-us dir=ltr align=left>
<HR tabIndex=-1>
<FONT face=Tahoma size=2><B>From:</B> openib-general-bounces@openib.org
[mailto:openib-general-bounces@openib.org] <B>On Behalf Of </B>Michael
Krause<BR><B>Sent:</B> Wednesday, November 09, 2005 12:21 PM<BR><B>To:</B>
Rick Frank; Ranjit Pandit<BR><B>Cc:</B>
openib-general@openib.org<BR><B>Subject:</B> Re: [openib-general] [ANNOUNCE]
Contribute RDS(ReliableDatagramSockets) to OpenIB<BR></FONT><BR></DIV>
<DIV></DIV>
<BLOCKQUOTE class=cite cite="" type="cite">One could be able to talk to the
remote node across other HCA but that does not mean one has an understanding
of the state at the remote node unless the failure is noted and a resync of
state occurs or the remote is able to deal with duplicates, etc.
This has nothing to do with API or the transport involved but, as Caitlin
noted, the difference between knowing a send buffer is free vs. knowing that
the application received the data requested. Therefore, one has only
reduced the reliability / robustness problem space to some extent but has
not solved it by the use of RDS.<BR><BR></BLOCKQUOTE></BLOCKQUOTE>
<DIV dir=ltr><FONT face=Arial color=#0000ff size=2><SPAN
class=614573020-09112005>Correct. When there are point-to-point credits (even if
only enforced/understood</SPAN></FONT></DIV>
<DIV dir=ltr><FONT face=Arial color=#0000ff size=2><SPAN
class=614573020-09112005>at the ULP) then the application can correctly infer
that message N was</SPAN></FONT></DIV>
<DIV dir=ltr><FONT face=Arial color=#0000ff size=2><SPAN
class=614573020-09112005>successfully processed because the matching credit was
restored. A transport</SPAN></FONT></DIV>
<DIV dir=ltr><FONT face=Arial color=#0000ff size=2><SPAN
class=614573020-09112005>neutral application can only communicate restoration of
credits via ULP</SPAN></FONT></DIV>
<DIV dir=ltr><FONT face=Arial color=#0000ff size=2><SPAN
class=614573020-09112005>messaging. When credits are shared across sessions then
the ULP</SPAN></FONT></DIV>
<DIV dir=ltr><FONT face=Arial color=#0000ff size=2><SPAN
class=614573020-09112005>has a much more complex task to properly communicate
credits.</SPAN></FONT></DIV>
<DIV dir=ltr><FONT face=Arial color=#0000ff size=2><SPAN
class=614573020-09112005></SPAN></FONT> </DIV>
<DIV dir=ltr><FONT face=Arial color=#0000ff size=2><SPAN
class=614573020-09112005>The proposal I presented at RAIT for multistreamed MPA
had a non-highlighted</SPAN></FONT></DIV>
<DIV dir=ltr><FONT face=Arial color=#0000ff size=2><SPAN
class=614573020-09112005>option for a "wildcard" endpoint. Without the option
multistream MPA is essentially</SPAN></FONT></DIV>
<DIV dir=ltr><FONT face=Arial color=#0000ff size=2><SPAN
class=614573020-09112005>the SCTP adaptation for RDMA running over plain
MPA/TCP. It achieves the</SPAN></FONT></DIV>
<DIV dir=ltr><FONT face=Arial color=#0000ff size=2><SPAN
class=614573020-09112005>same reduction in reliable transport layer connections
that RDS does, but</SPAN></FONT></DIV>
<DIV dir=ltr><FONT face=Arial color=#0000ff size=2><SPAN
class=614573020-09112005>does not reduce the number of RDMA endpoints. The
wildcard option </SPAN></FONT></DIV>
<DIV dir=ltr><FONT face=Arial color=#0000ff size=2><SPAN
class=614573020-09112005>reduces the number of RDMA endpoints as well, but
greatly complicates</SPAN></FONT></DIV>
<DIV dir=ltr><FONT face=Arial color=#0000ff size=2><SPAN
class=614573020-09112005>the RDMA state machines. RDS over IB faces similar
problems, but solved</SPAN></FONT></DIV>
<DIV dir=ltr><FONT face=Arial color=#0000ff size=2><SPAN
class=614573020-09112005>them slightly differently.</SPAN></FONT></DIV>
<DIV dir=ltr><FONT face=Arial color=#0000ff size=2><SPAN
class=614573020-09112005></SPAN></FONT> </DIV>
<DIV dir=ltr><FONT face=Arial color=#0000ff size=2><SPAN
class=614573020-09112005>Over iWARP I believe these complexities favor keeping
the point-to-point</SPAN></FONT></DIV>
<DIV dir=ltr><FONT face=Arial color=#0000ff size=2><SPAN
class=614573020-09112005>logical connection between QP and only reducing the
number of L4 </SPAN></FONT></DIV>
<DIV dir=ltr><FONT face=Arial color=#0000ff size=2><SPAN
class=614573020-09112005>connections (from many TCP connections to a single TCP
connection</SPAN></FONT></DIV>
<DIV dir=ltr><FONT face=Arial color=#0000ff size=2><SPAN
class=614573020-09112005>or SCTP association). The advantage of that approach is
that the API</SPAN></FONT></DIV>
<DIV dir=ltr><FONT face=Arial color=#0000ff size=2><SPAN
class=614573020-09112005>from application to RDMA endpoint (QP) can be left
totally unchanged.</SPAN></FONT></DIV>
<DIV dir=ltr><FONT face=Arial color=#0000ff size=2><SPAN
class=614573020-09112005>But I do not see any such option over IB, unless RD is
improved or a</SPAN></FONT></DIV>
<DIV dir=ltr><FONT face=Arial color=#0000ff size=2><SPAN
class=614573020-09112005>new SCTP-like connection mode is
defined.</SPAN></FONT></DIV>
<DIV dir=ltr><FONT face=Arial color=#0000ff size=2><SPAN
class=614573020-09112005></SPAN></FONT> </DIV>
<DIV dir=ltr><FONT face=Arial color=#0000ff size=2><SPAN
class=614573020-09112005>In my opinion the multi-streaming is the most important
feature here,</SPAN></FONT></DIV>
<DIV dir=ltr><FONT face=Arial color=#0000ff size=2><SPAN
class=614573020-09112005>but over IB I do not think there is a natural
adaptation that provides</SPAN></FONT></DIV>
<DIV dir=ltr><FONT face=Arial color=#0000ff size=2><SPAN
class=614573020-09112005>multi-streaming without also adding the any-to-any
endpoint semantics.</SPAN></FONT></DIV>
<DIV dir=ltr><FONT face=Arial color=#0000ff size=2><SPAN
class=614573020-09112005>Multistream MPA and SCTP can both support the
any-to-any endpoint</SPAN></FONT></DIV>
<DIV dir=ltr><FONT face=Arial color=#0000ff size=2><SPAN
class=614573020-09112005>semantics by moving the source to payload information
rather than</SPAN></FONT></DIV>
<DIV dir=ltr><FONT face=Arial color=#0000ff size=2><SPAN
class=614573020-09112005>transport information (by invoking "wildcard status" in
MS-MPA or</SPAN></FONT></DIV>
<DIV dir=ltr><FONT face=Arial color=#0000ff size=2><SPAN
class=614573020-09112005>by duplicating the field for SCTP). So the RDS API
strikes me as</SPAN></FONT></DIV>
<DIV dir=ltr><FONT face=Arial color=#0000ff size=2><SPAN
class=614573020-09112005>the best option for a transport neutral application.
MS-MPA and SCTP</SPAN></FONT></DIV>
<DIV dir=ltr><FONT face=Arial color=#0000ff size=2><SPAN
class=614573020-09112005>reductions in transport overhead would be available
without special</SPAN></FONT></DIV>
<DIV dir=ltr><FONT face=Arial color=#0000ff size=2><SPAN
class=614573020-09112005>API support.</SPAN></FONT></DIV>
<DIV dir=ltr><FONT face=Arial color=#0000ff size=2><SPAN
class=614573020-09112005></SPAN></FONT> </DIV></BODY></HTML>