<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META http-equiv=Content-Type content="text/html; charset=us-ascii">
<META content="MSHTML 6.00.2900.2722" name=GENERATOR></HEAD>
<BODY>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff
size=2></FONT> </DIV><BR>
<BLOCKQUOTE
style="PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT: #0000ff 2px solid; MARGIN-RIGHT: 0px">
<DIV class=OutlookMessageHeader lang=en-us dir=ltr align=left>
<HR tabIndex=-1>
</DIV>
<DIV class=OutlookMessageHeader lang=en-us dir=ltr align=left><FONT
face=Tahoma><FONT size=2><B>From:</B> openib-general-bounces@openib.org
[mailto:openib-general-bounces@openib.org] <B>On Behalf Of </B>Michael
Krause<BR><B>Sent:</B> Tuesday, November 08, 2005 11:52 AM<BR><B>To:</B>
Rimmer, Todd<BR><B>Cc:</B> openib-general@openib.org<BR><B>Subject:</B> RE:
[openib-general] [ANNOUNCE] Contribute RDS ( ReliableDatagramSockets) to
OpenIB<BR><SPAN class=553424220-08112005><FONT face=Arial
color=#0000ff></FONT></SPAN></FONT></FONT></DIV>
<DIV class=OutlookMessageHeader lang=en-us dir=ltr align=left><FONT
face=Tahoma><FONT size=2><SPAN
class=553424220-08112005> </SPAN></FONT></FONT><FONT size=3>The entire
discussion might be distilled into the following:<BR><BR>- Datagram
applications trade reliability for flexibility and resource savings.
<BR><BR></FONT></DIV></BLOCKQUOTE>
<DIV class=OutlookMessageHeader lang=en-us align=left><FONT face=Arial
color=#0000ff size=2><SPAN class=553424220-08112005>Reliable Datagram
applications have endpoints that accept messages from
multiple</SPAN></FONT></DIV>
<DIV class=OutlookMessageHeader lang=en-us align=left><FONT face=Arial
color=#0000ff size=2><SPAN class=553424220-08112005>known sources, rather than
from a single known source (TCP, RC) or multiple </SPAN></FONT></DIV>
<DIV class=OutlookMessageHeader lang=en-us align=left><FONT face=Arial
color=#0000ff size=2><SPAN class=553424220-08112005>unknown sources (UDP,
RD).</SPAN></FONT></DIV>
<DIV class=OutlookMessageHeader lang=en-us align=left><FONT face=Arial
color=#0000ff size=2><SPAN class=553424220-08112005></SPAN></FONT> </DIV>
<DIV class=OutlookMessageHeader lang=en-us align=left><FONT face=Arial
color=#0000ff size=2><SPAN class=553424220-08112005>This does save resources,
but perhaps just as importantly it may reflect how</SPAN></FONT></DIV>
<DIV class=OutlookMessageHeader lang=en-us align=left><FONT face=Arial
color=#0000ff size=2><SPAN class=553424220-08112005>the application truly thinks
of its communication endpoints. Oracle is not unique</SPAN></FONT></DIV>
<DIV class=OutlookMessageHeader lang=en-us align=left><FONT face=Arial
color=#0000ff size=2><SPAN class=553424220-08112005>in this communication
requirement. This is essentially the interface MPI presents</SPAN></FONT></DIV>
<DIV class=OutlookMessageHeader lang=en-us align=left><FONT face=Arial
color=#0000ff size=2><SPAN class=553424220-08112005>to its users as
well.</SPAN></FONT></DIV>
<BLOCKQUOTE
style="PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT: #0000ff 2px solid; MARGIN-RIGHT: 0px">
<DIV class=OutlookMessageHeader lang=en-us dir=ltr align=left><FONT face=Arial
color=#0000ff size=2><SPAN class=553424220-08112005></SPAN></FONT> </DIV>
<DIV class=OutlookMessageHeader lang=en-us dir=ltr align=left><FONT
size=3><SPAN class=553424220-08112005> </SPAN><BR>- Datagram applications
that require reliability have to re-invent the wheel and given it is
non-trivial, they often get it variable quality and can suffer performance
loss if done poorly or the network is very lossy. Given networks are a
lot less lossy today than years past, sans congestion drops, one might argue
about whether there is still a significant problem or not.<BR><BR><SPAN
class=553424220-08112005><FONT face=Arial color=#0000ff
size=2>[cait] </FONT></SPAN></FONT></DIV></BLOCKQUOTE>
<DIV class=OutlookMessageHeader lang=en-us align=left><FONT face=Arial><FONT
size=2><FONT color=#0000ff><FONT size=3><SPAN
class=553424220-08112005>Standardized congestion control that is not dependent
on application specific</SPAN></FONT></FONT></FONT></FONT></DIV>
<DIV class=OutlookMessageHeader lang=en-us align=left><FONT face=Arial><FONT
size=2><FONT color=#0000ff><FONT size=3><SPAN class=553424220-08112005>control
is highly desirable. In the IP world new ULPs based upon UDP
are</SPAN></FONT></FONT></FONT></FONT></DIV>
<DIV class=OutlookMessageHeader lang=en-us align=left><FONT face=Arial><FONT
size=2><FONT color=#0000ff><FONT size=3><SPAN class=553424220-08112005>heavily
discouraged for exactly this reason.</SPAN></FONT></FONT></FONT></FONT></DIV>
<BLOCKQUOTE
style="PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT: #0000ff 2px solid; MARGIN-RIGHT: 0px">
<DIV class=OutlookMessageHeader lang=en-us dir=ltr align=left><FONT face=Arial
color=#0000ff size=2><SPAN class=553424220-08112005></SPAN></FONT> </DIV>
<DIV class=OutlookMessageHeader lang=en-us dir=ltr align=left><FONT
size=3><SPAN class=553424220-08112005> </SPAN><BR>- The reliable datagram
model isn't new - been there, done that on earlier interconnects - but it
isn't free. IB could have done something like RDS but the people who
pushed the original requirements (some who are advocating RDS now) did not
want to take on the associated software enablement thus it was subsumed into
hardware and made slightly more restrictive as a result - perhaps more than
some people may like. The only real delta between RDS one sense and the
current IB RD is the number of outstanding messages in flight on a given
EEC. If RD were re-defined to allow software to recover some types of
failures much like UC, then one could simply use RD.<BR><BR><SPAN
class=553424220-08112005><FONT face=Arial color=#0000ff
size=2>[cait] </FONT></SPAN></FONT></DIV></BLOCKQUOTE>
<DIV class=OutlookMessageHeader lang=en-us align=left><FONT face=Arial><FONT
size=2><FONT color=#0000ff><FONT size=3><SPAN class=553424220-08112005>The RDS
API should definitely be compatiable with IB RD service, especially any
later</SPAN></FONT></FONT></FONT></FONT></DIV>
<DIV class=OutlookMessageHeader lang=en-us align=left><FONT face=Arial><FONT
size=2><FONT color=#0000ff><FONT size=3><SPAN class=553424220-08112005>one that
solves the crippling limitation on in-flight
messages.</SPAN></FONT></FONT></FONT></FONT></DIV>
<DIV class=OutlookMessageHeader lang=en-us align=left><FONT face=Arial><FONT
size=2><FONT color=#0000ff><FONT size=3><SPAN
class=553424220-08112005></SPAN></FONT></FONT></FONT></FONT> </DIV>
<DIV class=OutlookMessageHeader lang=en-us align=left><FONT face=Arial><FONT
size=2><FONT color=#0000ff><FONT size=3><SPAN class=553424220-08112005>Similarly
the API should be compatible with IP based solutions, which since it is
derived</SPAN></FONT></FONT></FONT></FONT></DIV>
<DIV class=OutlookMessageHeader lang=en-us align=left><FONT face=Arial><FONT
size=2><FONT color=#0000ff><FONT size=3><SPAN class=553424220-08112005>from
SOCK_DGRAM isn't much of a challenge.</SPAN></FONT></FONT></FONT></FONT></DIV>
<DIV class=OutlookMessageHeader lang=en-us align=left><FONT face=Arial><FONT
size=2><FONT color=#0000ff><FONT size=3><SPAN
class=553424220-08112005></SPAN></FONT></FONT></FONT></FONT> </DIV>
<BLOCKQUOTE
style="PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT: #0000ff 2px solid; MARGIN-RIGHT: 0px">
<DIV class=OutlookMessageHeader lang=en-us dir=ltr align=left><FONT face=Arial
color=#0000ff size=2><SPAN class=553424220-08112005></SPAN></FONT> </DIV>
<DIV class=OutlookMessageHeader lang=en-us dir=ltr align=left><FONT
size=3><SPAN class=553424220-08112005> </SPAN><BR>- RDS does not solve a
set of failure models. For example, if a RNIC / HCA were to fail, then
one cannot simply replay the operations on another RNIC / HCA without
extracting state, etc. and providing some end-to-end sync of what was really
sent / received by the application. Yes, one can recover from cable or
switch port failure by using APM style recovery but that is only one class of
faults. The harder faults either result in the end node being cast out
of the cluster or see silent data corruption unless additional steps are taken
to transparently recover - again app writers don't want to solve the hard
problems; they want that done for them.<BR><SPAN
class=553424220-08112005><FONT face=Arial color=#0000ff
size=2>[cait] </FONT></SPAN></FONT></DIV></BLOCKQUOTE>
<DIV class=OutlookMessageHeader lang=en-us align=left><FONT face=Arial><FONT
size=2><FONT color=#0000ff><FONT size=3><SPAN class=553424220-08112005>This goes
to the question of where the Reliable Datagram Service is
implemented.</SPAN></FONT></FONT></FONT></FONT></DIV>
<DIV class=OutlookMessageHeader lang=en-us align=left><FONT face=Arial><FONT
size=2><FONT color=#0000ff><FONT size=3><SPAN class=553424220-08112005>When done
as middleware over existing reliable connection services then the
middleware</SPAN></FONT></FONT></FONT></FONT></DIV>
<DIV class=OutlookMessageHeader lang=en-us align=left><FONT face=Arial><FONT
size=2><FONT color=#0000ff><FONT size=3><SPAN class=553424220-08112005>does have
a few issues on handling flushed buffers after an RNIC failure. These issues
make</SPAN></FONT></FONT></FONT></FONT></DIV>
<DIV class=OutlookMessageHeader lang=en-us align=left><FONT face=Arial><FONT
size=2><FONT color=#0000ff><FONT size=3><SPAN
class=553424220-08112005>implementation of a zero-copy strategy more of an
issue.</SPAN></FONT></FONT></FONT></FONT></DIV>
<DIV class=OutlookMessageHeader lang=en-us align=left><FONT face=Arial><FONT
size=2><FONT color=#0000ff><FONT size=3><SPAN
class=553424220-08112005></SPAN></FONT></FONT></FONT></FONT> </DIV>
<DIV class=OutlookMessageHeader lang=en-us align=left><FONT face=Arial><FONT
size=2><FONT color=#0000ff><FONT size=3><SPAN class=553424220-08112005>But if
the endpoint is truly a datagram endpoint then these issues are the same as
for</SPAN></FONT></FONT></FONT></FONT></DIV>
<DIV class=OutlookMessageHeader lang=en-us align=left><FONT face=Arial><FONT
size=2><FONT color=#0000ff><FONT size=3><SPAN class=553424220-08112005>failover
of connection-oriented endpoints between two
RNICs/HCAs.</SPAN></FONT></FONT></FONT></FONT></DIV>
<DIV class=OutlookMessageHeader lang=en-us align=left><FONT face=Arial><FONT
size=2><FONT color=#0000ff><FONT size=3><SPAN
class=553424220-08112005></SPAN></FONT></FONT></FONT></FONT> </DIV>
<BLOCKQUOTE
style="PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT: #0000ff 2px solid; MARGIN-RIGHT: 0px">
<DIV class=OutlookMessageHeader lang=en-us dir=ltr align=left><FONT
size=3><SPAN class=553424220-08112005> </SPAN><BR><BR>- RNIC / HCA
provide hardware acceleration and reliable delivery to the remote RNIC / HCA
(not to the application since that is in a separate fault domain). Doing
software multiplexing over such an interconnect as envisioned for IB RD is
relatively straight in many respects but not a trivial exercise as some might
contend. Yes, people can point to a small number of lines of code but
that is just for the initial offering and is not an indication of what it
might have to become long-term to add all of the bells-n-whistles that people
have envisioned.<BR><BR><SPAN class=553424220-08112005><FONT face=Arial
color=#0000ff size=2>[cait] </FONT></SPAN></FONT></DIV></BLOCKQUOTE>
<DIV class=OutlookMessageHeader lang=en-us align=left><FONT face=Arial><FONT
size=2><FONT color=#0000ff><FONT size=3><SPAN class=553424220-08112005>IB RD is
not transport neutral, and has the problem of severe in-flight limitations
that</SPAN></FONT></FONT></FONT></FONT></DIV>
<DIV class=OutlookMessageHeader lang=en-us align=left><FONT face=Arial><FONT
size=2><FONT color=#0000ff><FONT size=3><SPAN class=553424220-08112005>would
make it unacceptable to most applications that would benefit from RDS
even</SPAN></FONT></FONT></FONT></FONT></DIV>
<DIV class=OutlookMessageHeader lang=en-us align=left><FONT face=Arial><FONT
size=2><FONT color=#0000ff><FONT size=3><SPAN class=553424220-08112005>if they
were </SPAN></FONT></FONT></FONT></FONT></DIV>
<DIV class=OutlookMessageHeader lang=en-us align=left><FONT face=Arial><FONT
size=2><FONT color=#0000ff><FONT size=3><SPAN
class=553424220-08112005></SPAN></FONT></FONT></FONT></FONT> </DIV>
<DIV class=OutlookMessageHeader lang=en-us align=left><FONT face=Arial><FONT
size=2><FONT color=#0000ff><FONT size=3><SPAN class=553424220-08112005>There is
no way that iWARP vendors would ever implement a service designed
to</SPAN></FONT></FONT></FONT></FONT></DIV>
<DIV class=OutlookMessageHeader lang=en-us align=left><FONT face=Arial><FONT
size=2><FONT color=#0000ff><FONT size=3><SPAN class=553424220-08112005>match IB
RD. An RDS service could be implemented over TCP, MPA,
MS-MPA</SPAN></FONT></FONT></FONT></FONT></DIV>
<DIV class=OutlookMessageHeader lang=en-us align=left><FONT face=Arial><FONT
size=2><FONT color=#0000ff><FONT size=3><SPAN class=553424220-08112005>or
SCTP.</SPAN></FONT></FONT></FONT></FONT></DIV>
<DIV class=OutlookMessageHeader lang=en-us align=left><FONT face=Arial><FONT
size=2><FONT color=#0000ff><FONT size=3><SPAN
class=553424220-08112005></SPAN></FONT></FONT></FONT></FONT> </DIV>
<BLOCKQUOTE
style="PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT: #0000ff 2px solid; MARGIN-RIGHT: 0px">
<DIV class=OutlookMessageHeader lang=en-us dir=ltr align=left><FONT
size=3><SPAN class=553424220-08112005> </SPAN><BR>- RDS is not an API but
a ULP. It really uses a set of physical connections and which are then
used to set up logical application associations (often referred to as
connections but really are not in terms of the interconnect). These
associations can be quickly established as they are just control messages over
the existing physical connections. Again, builds on concepts already
shipping in earlier interconnects / solutions from a number of years
back. Hence, for large scale applications which are association
intensive, RDS is able to improve the performance of establishing these
associations. While RDS improves the performance in this regard, its
impacts on actual performance stem more from avoiding some operations thus
nearly all of the performance numbers quoted are really an apple-to-orange
comparison. Nothing wrong with this but people need to keep in mind that
things are not being compared with one another on the same level thus the
results can look more dramatic.<BR><SPAN class=553424220-08112005><FONT
face=Arial color=#0000ff
size=2>[cait] </FONT></SPAN></FONT></DIV></BLOCKQUOTE>
<DIV class=OutlookMessageHeader lang=en-us align=left><FONT face=Arial><FONT
size=2><FONT color=#0000ff><FONT size=3><SPAN class=553424220-08112005>All
correct.</SPAN></FONT></FONT></FONT></FONT></DIV>
<DIV class=OutlookMessageHeader lang=en-us align=left><FONT face=Arial><FONT
size=2><FONT color=#0000ff><FONT size=3><SPAN
class=553424220-08112005></SPAN></FONT></FONT></FONT></FONT> </DIV>
<DIV class=OutlookMessageHeader lang=en-us align=left><FONT face=Arial><FONT
size=2><FONT color=#0000ff><FONT size=3><SPAN class=553424220-08112005>The real
issue with RDS is whether it makes sense to present this a pseudo-transport
service,</SPAN></FONT></FONT></FONT></FONT></DIV>
<DIV class=OutlookMessageHeader lang=en-us align=left><FONT face=Arial><FONT
size=2><FONT color=#0000ff><FONT size=3><SPAN class=553424220-08112005>or if its
just a suggested strategy that each application should implement on its own.
>From a</SPAN></FONT></FONT></FONT></FONT></DIV>
<DIV class=OutlookMessageHeader lang=en-us align=left><FONT face=Arial><FONT
size=2><FONT color=#0000ff><FONT size=3><SPAN class=553424220-08112005>wire
perspective there isn't much different. From a development perspective it makes
sense</SPAN></FONT></FONT></FONT></FONT></DIV>
<DIV class=OutlookMessageHeader lang=en-us align=left><FONT face=Arial><FONT
size=2><FONT color=#0000ff><FONT size=3><SPAN class=553424220-08112005>as long
as the pseudo-transport definition is indeed defined as though it were a
transport.</SPAN></FONT></FONT></FONT></FONT></DIV>
<DIV class=OutlookMessageHeader lang=en-us align=left><FONT face=Arial><FONT
size=2><FONT color=#0000ff><FONT size=3><SPAN class=553424220-08112005>I believe
that is the case here.</SPAN></FONT></FONT></FONT></FONT></DIV>
<DIV class=OutlookMessageHeader lang=en-us align=left><FONT face=Arial><FONT
size=2><FONT color=#0000ff><FONT size=3><SPAN
class=553424220-08112005></SPAN></FONT></FONT></FONT></FONT> </DIV>
<BLOCKQUOTE
style="PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT: #0000ff 2px solid; MARGIN-RIGHT: 0px">
<DIV class=OutlookMessageHeader lang=en-us dir=ltr align=left><FONT
size=3><SPAN class=553424220-08112005> </SPAN><BR><BR>- One thing to keep
in mind is that RDS is about not doing work to gain performance and to
potentially improve code by eliminating software that was too complex /
difficult to get clean when it was invoked to recover from fabric-related
issues. This is somewhat the same logic as used by NFS when migrating to
TCP from UDP. Could not get clean software so change the
underlying comms to push the problem to a place where it is largely
solved.<BR><BR>Now, whether you believe RDS is great or not, it is an attempt
to solve a problem plaguing one class of applications who'd rather not spend
their resources on the problem. That is a fair thing to consider if
someone else has already done it better using another technology. One
could also consider having IB change the RD semantics to see if that would
solve the problem since it would not require a new ULP to make it work when
you think about it though there is no analog with iWARP. The discussion
so far has been interesting and I think there is fair push back to avoid
re-inventing the wheel especially on the idea of trying to do this directly on
Ethernet (that seems like just re-inventing all of that buggy code people
stated they could not get right at the app layer in the first place and
largely goes against the logic used to create IB and as well as iWARP's use of
TCP in the first place).<BR><BR><BR></FONT><SPAN
class=553424220-08112005><FONT face=Arial color=#0000ff
size=2>[cait] </FONT></SPAN></DIV></BLOCKQUOTE>
<DIV class=OutlookMessageHeader lang=en-us align=left><FONT face=Arial><FONT
size=2><FONT color=#0000ff><SPAN class=553424220-08112005>If there were a
definition of a usable RD service from IB then porting it to iWARP
could</SPAN></FONT></FONT></FONT></DIV>
<DIV class=OutlookMessageHeader lang=en-us align=left><FONT face=Arial><FONT
size=2><FONT color=#0000ff><SPAN class=553424220-08112005>be considered as well.
The key characteristics are a) message orientation, b)
reliable</SPAN></FONT></FONT></FONT></DIV>
<DIV class=OutlookMessageHeader lang=en-us align=left><FONT face=Arial><FONT
size=2><FONT color=#0000ff><SPAN class=553424220-08112005>delivery c) multiple
known source (not unlimited unknown) and d) multiple
in-flight</SPAN></FONT></FONT></FONT></DIV>
<DIV class=OutlookMessageHeader lang=en-us align=left><FONT face=Arial><FONT
size=2><FONT color=#0000ff><SPAN class=553424220-08112005>messages with the ULP
being responsible for flow control.</SPAN></FONT></FONT></FONT></DIV>
<DIV class=OutlookMessageHeader lang=en-us align=left><FONT face=Arial><FONT
size=2><FONT color=#0000ff><SPAN
class=553424220-08112005></SPAN></FONT></FONT></FONT> </DIV>
<BLOCKQUOTE
style="PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT: #0000ff 2px solid; MARGIN-RIGHT: 0px">
<DIV class=OutlookMessageHeader lang=en-us dir=ltr align=left><SPAN
class=553424220-08112005><FONT face=Arial color=#0000ff
size=2></FONT></SPAN> </DIV>
<DIV class=OutlookMessageHeader lang=en-us dir=ltr align=left><SPAN
class=553424220-08112005><FONT face=Arial color=#0000ff
size=2></FONT></SPAN> </DIV>
<DIV class=OutlookMessageHeader lang=en-us dir=ltr align=left><SPAN
class=553424220-08112005> </SPAN></DIV></BLOCKQUOTE></BODY></HTML>