<html><body>
<p>Set default as 4 (2K) is more proper than 1(512?). All HCAs support 2K at least now. <br>
<br>
Thanks<br>
Shirley Ma<br>
<br>
<br>
<img width="16" height="16" src="cid:1__=08BBF9B7DFC1C67A8f9e8a93df938@us.ibm.com" border="0" alt="Inactive hide details for "Michael S. Tsirkin" <mst@dev.mellanox.co.il>">"Michael S. Tsirkin" <mst@dev.mellanox.co.il><br>
<br>
<br>
<table width="100%" border="0" cellspacing="0" cellpadding="0">
<tr valign="top"><td style="background-image:url(cid:2__=08BBF9B7DFC1C67A8f9e8a93df938@us.ibm.com); background-repeat: no-repeat; " width="40%">
<ul>
<ul>
<ul>
<ul><b><font size="2">"Michael S. Tsirkin" <mst@dev.mellanox.co.il></font></b><font size="2"> </font>
<p><font size="2">07/26/07 12:22 AM</font>
<table border="1">
<tr valign="top"><td width="168" bgcolor="#FFFFFF"><div align="center"><font size="2">Please respond to<br>
"Michael S. Tsirkin" <mst@dev.mellanox.co.il></font></div></td></tr>
</table>
</ul>
</ul>
</ul>
</ul>
</td><td width="60%">
<table width="100%" border="0" cellspacing="0" cellpadding="0">
<tr valign="top"><td width="1%"><img width="58" height="1" src="cid:3__=08BBF9B7DFC1C67A8f9e8a93df938@us.ibm.com" border="0" alt=""><br>
<div align="right"><font size="2">To</font></div></td><td width="100%"><img width="1" height="1" src="cid:3__=08BBF9B7DFC1C67A8f9e8a93df938@us.ibm.com" border="0" alt=""><br>
<font size="2">Shirley Ma/Beaverton/IBM@IBMUS</font></td></tr>
<tr valign="top"><td width="1%"><img width="58" height="1" src="cid:3__=08BBF9B7DFC1C67A8f9e8a93df938@us.ibm.com" border="0" alt=""><br>
<div align="right"><font size="2">cc</font></div></td><td width="100%"><img width="1" height="1" src="cid:3__=08BBF9B7DFC1C67A8f9e8a93df938@us.ibm.com" border="0" alt=""><br>
<font size="2">Eitan Zahavi <eitan@mellanox.co.il>, general@lists.openfabrics.org</font></td></tr>
<tr valign="top"><td width="1%"><img width="58" height="1" src="cid:3__=08BBF9B7DFC1C67A8f9e8a93df938@us.ibm.com" border="0" alt=""><br>
<div align="right"><font size="2">Subject</font></div></td><td width="100%"><img width="1" height="1" src="cid:3__=08BBF9B7DFC1C67A8f9e8a93df938@us.ibm.com" border="0" alt=""><br>
<font size="2">Re: Re: openSM: Different IB MTUs</font></td></tr>
</table>
<table border="0" cellspacing="0" cellpadding="0">
<tr valign="top"><td width="58"><img width="1" height="1" src="cid:3__=08BBF9B7DFC1C67A8f9e8a93df938@us.ibm.com" border="0" alt=""></td><td width="336"><img width="1" height="1" src="cid:3__=08BBF9B7DFC1C67A8f9e8a93df938@us.ibm.com" border="0" alt=""></td></tr>
</table>
</td></tr>
</table>
<br>
<tt>What does "1" mean? Surely not 1 byte MTU :)<br>
IMO a good format would be the MTU value in bytes.<br>
E.g. 512, 1024, 2048, 4096.<br>
<br>
Quoting Shirley Ma <xma@us.ibm.com>:<br>
Subject: RE: Re: openSM: Different IB MTUs<br>
<br>
Eitan,<br>
<br>
That's a good approach to address the issue.<br>
<br>
thanks<br>
Shirley Ma<br>
<br>
Inactive hide details for "Eitan Zahavi" <eitan@mellanox.co.il>"Eitan Zahavi"<br>
<eitan@mellanox.co.il><br>
<br>
<br>
"Eitan Zahavi" [cid] *<br>
<eitan@mellanox.co.il> To Shirley Ma/Beaverton/IBM@IBMUS<br>
[cid] *<br>
07/25/07 11:00 PM cc <general@lists.openfabrics.org>, "Hal Rosenstock"<br>
<hal.rosenstock@gmail.com><br>
[cid] *<br>
Subject RE: [ofa-general] Re: openSM: Different IB MTUs<br>
* *<br>
<br>
I propose that when there is no MTU in the partition policy file OpenSM use a<br>
configurable default from: /etc/cache/opensm/opensm.opt.<br>
Something like:<br>
# The default MTU to be used for IPoIB and other MCGs when the partition-policy<br>
# does not provide exact value. The default is the lowest possible MTU<br>
mcg_default_mtu 1<br>
<br>
Eitan Zahavi<br>
Senior Engineering Director, Software Architect<br>
Mellanox Technologies LTD<br>
Tel:+972-4-9097208<br>
Fax:+972-4-9593245<br>
P.O. Box 586 Yokneam 20692 ISRAEL<br>
<br>
<br>
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━<br>
From: Shirley Ma [<a href="mailto:xma@us.ibm.com">mailto:xma@us.ibm.com</a>]<br>
Sent: Wednesday, July 25, 2007 10:45 PM<br>
To: Eitan Zahavi<br>
Cc: general@lists.openfabrics.org; Hal Rosenstock<br>
Subject: RE: [ofa-general] Re: openSM: Different IB MTUs<br>
<br>
Hello Eitan, Hal,<br>
<br>
Thanks. It's good openSM has the configuration option to set up these<br>
attributes in MC. Is this a good idea to add below to openSM: When there is no<br>
MTU defined in the configuration file, SM can pick up the smallest link MTU in<br>
the fabrics by default? MTU is unlikely rate, slower rate might indicate the<br>
cablling problem. So using the smallest link MTU in the fabrics might not be a<br>
bad choice for MC by default. The reason I request here is to create IP<br>
multicast group, MTU is not an attribute of the group. When mapping IP<br>
multicast to IB multicast, IB muliticast might fail because of different IB<br>
link MTU size in the group, but IP multicast group will be successful without<br>
knowing the failure. If admin sets MTU in configuration file, admin would know<br>
this failure. Otherwise, admin/users could spend too much time on debugging<br>
their broken multicasting applications.<br>
<br>
Thanks<br>
Shirley Ma<br>
<br>
Inactive hide details for "Eitan Zahavi" <eitan@mellanox.co.il>"Eitan Zahavi"<br>
<eitan@mellanox.co.il><br>
<br>
"Eitan Zahavi" [cid] *<br>
<eitan@mellanox.co.il> To "Hal Rosenstock"<br>
<hal.rosenstock@gmail.com>, Shirley<br>
07/25/07 12:25 PM Ma/Beaverton/IBM@IBMUS<br>
[cid] *<br>
cc <general@lists.openfabrics.org><br>
[cid] *<br>
Subject RE: [ofa-general] Re: openSM:<br>
Different IB MTUs<br>
* *<br>
<br>
Hi Shirley,<br>
<br>
I think I understand where your question comes from...<br>
Many have issue with heterogonous fabrics where not all nodes have same MTU or<br>
Speed.<br>
Especially when IPoIB relies on all nodes joining the broadcast group.<br>
<br>
The term "join" for multicast groups is a little overloaded.<br>
If a node joins an existing MC group it has to have a rate (speed * width) ><br>
MCG.rate and support MTU > MCG.MTU otherwise it is denied.<br>
If the join is actually a "create" the node has to provide the rate and MTU<br>
which define the MCG values.<br>
<br>
To allow for administrator to control the IPoIB MCGs MTU and rate OpenSM<br>
provides the means to control these<br>
values per partition. See the doc/partition-config.doc<br>
Still the administrator should know what would be the lowest MTU and rate the<br>
nodes expected to join the IPoIB subnet have.<br>
The tradeoff is in the hands of the administrator who can set a value that will<br>
prevent slow nodes from joining the group,<br>
or assign a low value that will fit all nodes but slow down communication ...<br>
<br>
EZ<br>
<br>
Eitan Zahavi<br>
Senior Engineering Director, Software Architect<br>
Mellanox Technologies LTD<br>
Tel:+972-4-9097208<br>
Fax:+972-4-9593245<br>
P.O. Box 586 Yokneam 20692 ISRAEL<br>
<br>
<br>
<br>
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━<br>
From: general-bounces@lists.openfabrics.org [<br>
<a href="mailto:general-bounces@lists.openfabrics.org">mailto:general-bounces@lists.openfabrics.org</a>] On Behalf Of Hal Rosenstock<br>
Sent: Wednesday, July 25, 2007 10:01 PM<br>
To: Shirley Ma<br>
Cc: general@lists.openfabrics.org<br>
Subject: [ofa-general] Re: openSM: Different IB MTUs<br>
<br>
Shirley,<br>
<br>
On 7/25/07, Shirley Ma <xma@us.ibm.com> wrote:<br>
<br>
Hal,<br>
<br>
Thanks for your prompt reply. I am asking for how openSM handle<br>
different link MTUs in SA MCMemberRecord MTU. For example, if we have<br>
some links MTU as 2K, some links MTU as 1K. Then when enabling IPoIB,<br>
how does SM decide IPoIB broadcast group MCMemberRecord MTU size? When<br>
creating an IB multicast group from a 2K MTU node first, which PMTU<br>
value is attaching to this IB multicast group MCMemberRecord MTU?<br>
<br>
<br>
<br>
MCMemberRecord MTU gets the group MTU (when created). This is either this first<br>
joiner with sufficient components or preconfigured (and MTU can be set in the<br>
config). If a joiner has insufficient MTU for the group, it is denied.<br>
<br>
-- Hal<br>
<br>
Thanks<br>
Shirley Ma<br>
<br>
Inactive hide details for "Hal Rosenstock" <hal.rosenstock@gmail.com><br>
"Hal Rosenstock" < hal.rosenstock@gmail.com><br>
"Hal Rosenstock" < [cid] *<br>
hal.rosenstock@gmail.com> To Shirley Ma/Beaverton/<br>
IBM@IBMUS<br>
07/25/07 10:57 AM [cid] *<br>
cc general@lists.openfabrics.org<br>
[cid] *<br>
Subject Re: openSM: Different IB MTUs<br>
* *<br>
<br>
Shirley,<br>
<br>
On 7/25/07, Shirley Ma < xma@us.ibm.com> wrote:<br>
Hello Hal,<br>
<br>
How does openSM handle CAs with different MTUs in the<br>
same subnet? For example, IPoIB broadcast group MTU, IB<br>
multicast group PMTU? Does openSM pick up the smallest<br>
MTU in the subnet?<br>
<br>
<br>
Are you asking about link MTU, SA PathRecord/MultiPathRecord MTU, SA<br>
MCMemberRecord MTU, or all of these ?<br>
<br>
-- Hal<br>
Thanks<br>
Shirley Ma<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
_______________________________________________<br>
general mailing list<br>
general@lists.openfabrics.org<br>
</tt><tt><a href="http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general">http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general</a></tt><tt><br>
<br>
To unsubscribe, please visit </tt><tt><a href="http://openib.org/mailman/listinfo/openib-general">http://openib.org/mailman/listinfo/openib-general</a></tt><tt><br>
<br>
-- <br>
MST<br>
</tt><br>
</body></html>