<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=us-ascii">
<TITLE>Message</TITLE>
<STYLE type=text/css>BLOCKQUOTE {
PADDING-BOTTOM: 0px; PADDING-TOP: 0px
}
DL {
PADDING-BOTTOM: 0px; PADDING-TOP: 0px
}
UL {
PADDING-BOTTOM: 0px; PADDING-TOP: 0px
}
OL {
PADDING-BOTTOM: 0px; PADDING-TOP: 0px
}
LI {
PADDING-BOTTOM: 0px; PADDING-TOP: 0px
}
</STYLE>
<META content="MSHTML 6.00.2900.2523" name=GENERATOR></HEAD>
<BODY>
<DIV><SPAN class=187521012-06012005><FONT face=Arial color=#0000ff size=2>I feel
like we are talking about different things here:</FONT></SPAN></DIV>
<DIV><SPAN class=187521012-06012005><FONT face=Arial color=#0000ff
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=187521012-06012005><FONT face=Arial color=#0000ff size=2>The
***IP*** MTU is relevant for IPoIB performance because it determines the number
of times that you are going to be hit by the per-packet overhead of the
***host*** networking stack. My point was that the ***IP MTU*** will not be tied
to the ***IB*** MTU if a connected mode IPoIB (or SDP) is used instead of
the current IPoIB that uses IB UD transport service. The IB MTU would then
be irrelevant to this discussion.</FONT></SPAN></DIV>
<DIV><SPAN class=187521012-06012005><FONT face=Arial color=#0000ff
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=187521012-06012005><FONT face=Arial color=#0000ff size=2>As for
the eventual 2G ***IP*** MTU limit, it still sounds more than reasonable to me.
I wouldn't mind if a 10TB file gets split into some IP packets up to 2GB?!?!?
each.</FONT></SPAN></DIV>
<DIV><SPAN class=187521012-06012005><FONT face=Arial color=#0000ff
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=187521012-06012005><FONT face=Arial color=#0000ff size=2>(With
the exception of the UD transport service where IB messages are limited to be
single packet), the choice of ***IB*** MTU and its impact on performance is a
completely unrelated issue. IB messages are split into packets and reassembled
by the HCA HW. So the per-IB-message overhead of the SW stack is
independent of the IB MTU. The choice of IB MTU may indeed affect performance
for other reasons but it is not immediately obvious that the largest available
IB MTU is the best option for all cases. For example, latency optimization of
small high priority packets under load may benefit from smaller IB MTUs (e.g.
256).</FONT></SPAN></DIV>
<DIV><SPAN class=187521012-06012005><FONT face=Arial color=#0000ff
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=187521012-06012005><FONT face=Arial color=#0000ff
size=2>Diego</FONT></SPAN></DIV>
<DIV><SPAN class=187521012-06012005></SPAN><SPAN class=187521012-06012005><FONT
face=Arial color=#0000ff size=2></FONT></SPAN> </DIV>
<BLOCKQUOTE dir=ltr
style="PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT: #0000ff 2px solid; MARGIN-RIGHT: 0px">
<DIV></DIV>
<DIV class=OutlookMessageHeader lang=en-us dir=ltr align=left><FONT
face=Tahoma size=2>-----Original Message-----<BR><B>From:</B> Stephen Poole
[mailto:spoole@lanl.gov] <BR><B>Sent:</B> Thursday, January 06, 2005 5:45
AM<BR><B>To:</B> Diego Crupnicoff<BR><B>Cc:</B>
'openib-general@openib.org'<BR><B>Subject:</B> RE: [openib-general] ip over ib
throughtput<BR><BR></FONT></DIV>
<DIV>Have you done any "load" analysis of a 2K .vs. 4K MTU ? Your analogy of
having 2G as a total message size is potentially flawed. You seem to assume
that 2G is the end-all in size, it is not. What about when you want to (down
the road) use IB for files in the 1-10TB in size. Granted, we can live with
2G, but it is not some nirvana number. Second, with the 2G limit on messages
sizes, only determines the upper bound in overall size, I could send 2G @
32bytes MTU. So, the question is, how much less of a system load/impact would
a 4K MTU be over a 2K MTU. Remember, even Ethernet finally decided to go to
Jumbo Frames, why, system impact and more. Remember HIPPI/GSN, the MTU was
64K, reason, system impact. The numbers I have seen running IPoIB really
impact the system.</DIV>
<DIV><BR></DIV>
<DIV>Steve...</DIV>
<DIV><BR></DIV>
<DIV>At 10:38 AM -0800 1/5/05, Diego Crupnicoff wrote:</DIV>
<BLOCKQUOTE cite="" type="cite"><FONT size=-1>Note however that the relevant
IB limit is the max ***message size*** which happens to be equal to the
***IB*** MTU for the current IPoIB (that runs on top of IB UD transport
service where IB messages are limited to a single
packet).</FONT><BR></BLOCKQUOTE>
<BLOCKQUOTE cite="" type="cite"><FONT size=-1>A connected mode IPoIB (that
runs on top of IB RC/UC transport service) would allow IB messages up to 2GB
long. That will allow for much larger (effectively as large as you may ever
dream of) ***IP*** MTUs, regardless of the underlying IB
MTU.</FONT><BR></BLOCKQUOTE>
<BLOCKQUOTE cite="" type="cite"><FONT size=-1>Diego</FONT><BR></BLOCKQUOTE>
<BLOCKQUOTE cite="" type="cite"><FONT size=-1>> -----Original
Message-----</FONT><BR><FONT size=-1>> From: Hal Rosenstock [</FONT><A
href="mailto:halr@voltaire.com"><FONT
size=-1>mailto:halr@voltaire.com</FONT></A><FONT size=-1>]</FONT><BR><FONT
size=-1>> Sent: Wednesday, January 05, 2005 2:21 PM</FONT><BR><FONT
size=-1>> To: Peter Buckingham</FONT><BR><FONT size=-1>> Cc:
openib-general@openib.org</FONT><BR><FONT size=-1>> Subject: Re:
[openib-general] ip over ib throughtput</FONT><BR><FONT
size=-1>></FONT><BR><FONT size=-1>></FONT><BR><FONT size=-1>> On
Wed, 2005-01-05 at 12:23, Peter Buckingham wrote:</FONT><BR><FONT
size=-1>> > stupid question: why are we limited to a 2K MTU for
IPoIB?</FONT><BR><FONT size=-1>></FONT><BR><FONT size=-1>> The IB max
MTU is 4K. The current HCAs support a max MTU of 2K.</FONT><BR><FONT
size=-1>></FONT><BR><FONT size=-1>> -- Hal</FONT><BR><FONT
size=-1>></FONT><BR><FONT size=-1>>
_______________________________________________</FONT><BR><FONT size=-1>>
openib-general mailing list</FONT><BR><FONT size=-1>>
openib-general@openib.org</FONT><BR><FONT size=-1>></FONT> <A
href="http://openib.org/mailman/listinfo/openib-"><FONT
size=-1>http://openib.org/mailman/listinfo/openib-</FONT></A><FONT
size=-1>> general</FONT><BR><FONT size=-1>></FONT><BR><FONT
size=-1>> To</FONT><BR><FONT size=-1>> unsubscribe, please
visit</FONT><BR><FONT size=-1>></FONT> <A
href="http://openib.org/mailman/listinfo/openib-general"><FONT
size=-1>http://openib.org/mailman/listinfo/openib-general</FONT></A><BR><FONT
size=-1>></FONT><BR></BLOCKQUOTE>
<BLOCKQUOTE cite=""
type="cite"><BR>_______________________________________________<BR>openib-general
mailing
list<BR>openib-general@openib.org<BR>http://openib.org/mailman/listinfo/openib-general<BR><BR>To
unsubscribe, please visit
http://openib.org/mailman/listinfo/openib-general</BLOCKQUOTE>
<DIV><BR></DIV>
<DIV><BR></DIV><X-SIGSEP><PRE>--
</PRE></X-SIGSEP>
<DIV>Steve Poole (spoole@lanl.gov)<X-TAB>
</X-TAB><X-TAB>
</X-TAB><X-TAB>
</X-TAB><X-TAB>
</X-TAB><X-TAB> </X-TAB>Office:
505.665.9662<BR>Los Alamos National
Laboratory<X-TAB>
</X-TAB><X-TAB>
</X-TAB><X-TAB>
</X-TAB><X-TAB>
</X-TAB><X-TAB>
</X-TAB>Cell: 505.699.3807<BR>CCN - Special Projects /
Advanced Development<X-TAB>
</X-TAB><X-TAB>
</X-TAB><X-TAB>
</X-TAB>Fax: 505.665.7793<BR>P.O. Box 1663, MS B255<BR>Los
Alamos, NM. 87545<BR>03149801S</DIV></BLOCKQUOTE></BODY></HTML>