<br><br>
<div><span class="gmail_quote">On 8/25/09, <b class="gmail_sendername">Ira Weiny</b> <<a onclick="return top.js.OpenExtLink(window,event,this)" href="mailto:weiny2@llnl.gov" target="_blank">weiny2@llnl.gov</a>> wrote:</span>
<blockquote class="gmail_quote" style="PADDING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT: #ccc 1px solid">On Tue, 25 Aug 2009 19:15:19 -0400<br>Hal Rosenstock <<a onclick="return top.js.OpenExtLink(window,event,this)" href="mailto:hal.rosenstock@gmail.com" target="_blank">hal.rosenstock@gmail.com</a>> wrote:<br>
<br>> On 8/24/09, Ira Weiny <<a onclick="return top.js.OpenExtLink(window,event,this)" href="mailto:weiny2@llnl.gov" target="_blank">weiny2@llnl.gov</a>> wrote:<br>><br>> > If I send a combined DR path with a start lid but an empty (0 length) DR<br>
> > path.<br>><br>><br>> Hop Count 0 ?<br><br>Yes<br><br>><br>><br>> > What is the expected behavior?<br>><br>><br>> Not sure what you mean by expected here. Are you referring to expectation<br>
> based on the spec ?<br>><br><br>yes<br><br>><br>> > I know this could be specified with LID routing, but I don't see anywhere<br>> > in<br>> > the specification which says this is an error.<br>
><br>><br>> I don't think it should be an error (certainly not for the form you are<br>> using LID routed part followed by a DR part) but a null DR part is a little<br>> funny/odd.<br><br>Yea I know. It turns out that the new iblinkinfo issues queries like this<br>
when it is removes recurses back from the last DR portion of the combined<br>route path. It only showed up as an error when using the -S <guid> option of<br>iblinkinfo with this new switch I have. Works fine with the old switches.<br>
<br>><br>> > I do however seem to have 2<br>> > different implementations on 2 different switches. For example:<br>> ><br>> > I have Switch A (Lid 1) and Switch B (Lid 7). I attempt to query PortInfo<br>
> > of<br>> > Port 1 of each switch using the LID followed by an empty DR path.<br>> ><br>> > 17:55:22 > ./smpquery -c portinfo 1 0 1<br>> > ibwarn: [21005] mad_rpc: _do_madrpc failed; dport (Lid 1)<br>
> > ./smpquery: iberror: failed: operation portinfo: port info query failed<br>><br>><br>> Is this a timeout ?<br><br>yes<br><br>16:26:25 > ./smpquery -e -c portinfo 1 0 1<br>ibwarn: [27150] _do_madrpc: retry 1 (timeout 1000 ms)<br>
ibwarn: [27150] _do_madrpc: retry 2 (timeout 1000 ms)<br>ibwarn: [27150] _do_madrpc: timeout after 3 retries, 3000 ms<br>ibwarn: [27150] mad_rpc: _do_madrpc failed; dport (Lid 1)<br>./smpquery: iberror: failed: operation portinfo: port info query failed<br>
<br><br>><br>><br>> > 17:55:31 > ./smpquery -c portinfo 7 0 1<br>> > # Port info: Lid 7 port 1<br>> > Mkey:............................0x0000000000000000<br>> > GidPrefix:.......................0x0000000000000000<br>
> > ...<br>> > <normal output snipped><br>> ><br>> > Detecting this special case in libibmad and turning the packet into a LID<br>> > routed one<br>><br>><br>> Ugh... Is this special case really needed ? I don't think the underlying<br>
> issue is understood sufficiently yet.<br><br>Well I just did it to prove that what I was doing would work with a "simple"<br>lid routed packet. Like I said it might be that this portid which is being<br>specified to libibmad by libibnetdisc is not valid. If that is true then<br>
libibnetdisc should detect when the DR path is empty and go back to LID routed<br>requests. That is a valid fix in my mind.</blockquote>
<div> </div>
<div>Sure; there's no real need for combined route when the DR path is empty but it should work (at least with switches).</div><br>
<blockquote class="gmail_quote" style="PADDING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT: #ccc 1px solid">><br>> > succeeds but I wonder if this is an error in the SMI?<br>><br>><br>> Switch SMI ? Is this a proprietary implementation ?<br>
><br><br>Yes I see the bug with 2 different vendors switches. One is managed and the<br>other is not. My "old" switches (3 different vendors) do not show this<br>behavior. (Just to be clear I now I have 5 switches in my 5 node cluster!<br>
;-)<br><br>><br>><br>> > I also notice this is an error on the HCA I am running from (lid 2).<br>><br>><br>> Is this HCA node OpenIB based ?<br><br>yes</blockquote>
<div> </div>
<div>If I recall correctly, there is something in the spec that makes combined routing not be allowed on HCA (and router) ports so this seems correct. I can dig this out if really needed.</div><br>
<blockquote class="gmail_quote" style="PADDING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT: #ccc 1px solid">><br>> 17:57:42 > ./smpquery -c portinfo 2 0 1<br>> > ibwarn: [21008] mad_rpc: _do_madrpc failed; dport (Lid 2)<br>
> > ./smpquery: iberror: failed: operation portinfo: port info query failed<br>><br>><br>> Is this also a timeout ?<br><br>yes<br><br>><br>> Also, does the result differ based on where you source these from matter<br>
> (locally v. remotely)?<br><br>Same result local and remote.<br><br>><br>><br>><br>> > Running with a simple DR path works,<br>><br>><br>> You're referring to the same DR path here that fails in the combined route<br>
> examples above, right ?<br>><br><br>No. the example below is a DR path with Hop Count == 0 but without the initial<br>LID routing.<br><br>><br>> > I guess because this is the loopback case mentioned on page 805.<br>
><br>><br>> Yes but that's the high level requirement rather than the SMI rules which<br>> make that work.<br>><br>><br>><br>> > 17:58:16 > ./smpquery -D portinfo 0 1<br>> > # Port info: DR path slid 65535; dlid 65535; 0 port 1<br>
> > Mkey:............................0x0000000000000000<br>> > GidPrefix:.......................0x2007000000000000<br>> > ...<br>> > <snip><br>> ><br>> > It guess that the comment "Since each part may be empty, there are eight<br>
> > combinations, although only four are really useful:" on line 36 Page 805<br>> > can<br>> > be interpreted to mean that only those 4 combinations need to be supported.<br>> > Is this true?<br>
><br>><br>> Not all 4 combinations are supported/known to work. When this was added for<br>> ibportstate, the only combined routing form that was important was LID<br>> routed part followed by a DR part.<br>
><br><br>When you say "known to work" you mean implemented with the diags? Or known to<br>work in all hardware?</blockquote>
<div> </div>
<div>The former with most hardware up to some time ago. Note there is no compliance testing of combined routing and heavy reliance on this makes some a little nervous.</div><br>
<blockquote class="gmail_quote" style="PADDING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT: #ccc 1px solid">><br>> > On the other hand I think strictly this should be supported.<br>><br>><br>> In an ideal world yes but are they all required or is it just the one form<br>
> most heavily used ?<br><br>That is what I am unclear on. Does the spec require that all 8 combinations<br>are required to work? I don't see a specific compliance which says that and I<br>am not sure if C14-9 and C14-13 cover all 8 combinations.</blockquote>
<div> </div>
<div>I don't think there's any compliance on this. It all appears to be informative text. Perhaps a shortcoming of the spec. So there's nothing definitive. It just says there are 8 combinations (2**3 as there are 3 parts with 2 possibilities in each part) and that only 4 are really useful.</div>
<br>
<blockquote class="gmail_quote" style="PADDING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT: #ccc 1px solid">><br>> > Item 4 of C14-9<br>> > (line 24 page 810) requires the SMI to handle the packet if the HopPointer<br>
> > equals HopCount +1, which it is in my case (HopCount == 0, HopPointer == 1)<br>><br>><br>> By handle, this means "The SMI *shall *output the packet on the port whose<br>> number is in the entry indexed by Hop Pointer in the Initial Path. If that<br>
> port number is invalid, the SMI *shall *discard the SMP."<br>><br>> Are you sure the Hop Pointer is 1 ? Where do you see this ?<br><br>No I was wrong. I think I read the wrong madeye packet as I see the packet<br>
right before this one did have a hop pointer of 1. I Added some debug prints<br>to mad_encode to get the following output:<br><br>17:26:10 > ./smpquery -e -c portinfo 1 0 1<br>trid 2a0f0cb5; HopCount 0; HopPointer 0; slid 0; dlid 0; 0, drpath->cnt 0<br>
trid 2a0f0cb6; HopCount 0; HopPointer 0; slid 0; dlid 0; 0, drpath->cnt 0<br>trid 2a0f0cb7; HopCount 0; HopPointer 0; slid 2; dlid 65535; 0, drpath->cnt 0<br>ibwarn: [27322] _do_madrpc: recv failed: Connection timed out<br>
ibwarn: [27322] mad_rpc: _do_madrpc failed; dport (Lid 1)<br>./smpquery: iberror: failed: operation portinfo: port info query failed<br><br>madeye for these packets:<br><br>Aug 25 17:28:03 woprjr0 Madeye:recv SMP<br>Aug 25 17:28:03 woprjr0 MAD version....0x1<br>
Aug 25 17:28:03 woprjr0 Class..........0x81 (Directed route SMP)<br>Aug 25 17:28:03 woprjr0 Class version..0x1<br>Aug 25 17:28:03 woprjr0 Method.........0x81 (Get response)<br>Aug 25 17:28:03 woprjr0 Status.........0x8000<br>
Aug 25 17:28:03 woprjr0 Hop pointer....0x1<br>Aug 25 17:28:03 woprjr0 Hop counter....0x0<br>Aug 25 17:28:03 woprjr0 Trans ID.......0x1b9d2a0f0cb5<br>Aug 25 17:28:03 woprjr0 Attr ID........0x11 (node info)<br>Aug 25 17:28:03 woprjr0 Attr modifier..0x0000<br>
Aug 25 17:28:03 woprjr0 Mkey...........0x0<br>Aug 25 17:28:03 woprjr0 DR SLID........0xffff<br>Aug 25 17:28:03 woprjr0 DR DLID........0xffff<br>Aug 25 17:28:03 woprjr0 Madeye:sent SMP<br>Aug 25 17:28:03 woprjr0 MAD version....0x1<br>
Aug 25 17:28:03 woprjr0 Class..........0x81 (Directed route SMP)<br>Aug 25 17:28:03 woprjr0 Class version..0x1<br>Aug 25 17:28:03 woprjr0 Method.........0x1 (Get)<br>Aug 25 17:28:03 woprjr0 Status.........0x00<br>Aug 25 17:28:03 woprjr0 Hop pointer....0x1<br>
Aug 25 17:28:03 woprjr0 Hop counter....0x0<br>Aug 25 17:28:03 woprjr0 Trans ID.......0x1b9d2a0f0cb5<br>Aug 25 17:28:03 woprjr0 Attr ID........0x11 (node info)<br>Aug 25 17:28:03 woprjr0 Attr modifier..0x0000<br>Aug 25 17:28:03 woprjr0 Mkey...........0x0<br>
Aug 25 17:28:03 woprjr0 DR SLID........0xffff<br>Aug 25 17:28:03 woprjr0 DR DLID........0xffff<br>Aug 25 17:28:03 woprjr0 Madeye:recv SMP<br>Aug 25 17:28:03 woprjr0 MAD version....0x1<br>Aug 25 17:28:03 woprjr0 Class..........0x81 (Directed route SMP)<br>
Aug 25 17:28:03 woprjr0 Class version..0x1<br>Aug 25 17:28:03 woprjr0 Method.........0x81 (Get response)<br>Aug 25 17:28:03 woprjr0 Status.........0x8000<br>Aug 25 17:28:03 woprjr0 Hop pointer....0x1<br>Aug 25 17:28:03 woprjr0 Hop counter....0x0<br>
Aug 25 17:28:03 woprjr0 Trans ID.......0x1b9d2a0f0cb6<br>Aug 25 17:28:03 woprjr0 Attr ID........0x15 (port info)<br>Aug 25 17:28:03 woprjr0 Attr modifier..0x0000<br>Aug 25 17:28:03 woprjr0 Mkey...........0x0<br>Aug 25 17:28:03 woprjr0 DR SLID........0xffff<br>
Aug 25 17:28:03 woprjr0 DR DLID........0xffff<br>Aug 25 17:28:03 woprjr0 Madeye:sent SMP<br>Aug 25 17:28:03 woprjr0 MAD version....0x1<br>Aug 25 17:28:03 woprjr0 Class..........0x81 (Directed route SMP)<br>Aug 25 17:28:03 woprjr0 Class version..0x1<br>
Aug 25 17:28:03 woprjr0 Method.........0x1 (Get)<br>Aug 25 17:28:03 woprjr0 Status.........0x00<br>Aug 25 17:28:03 woprjr0 Hop pointer....0x1<br>Aug 25 17:28:03 woprjr0 Hop counter....0x0<br>Aug 25 17:28:03 woprjr0 Trans ID.......0x1b9d2a0f0cb6<br>
Aug 25 17:28:03 woprjr0 Attr ID........0x15 (port info)<br>Aug 25 17:28:03 woprjr0 Attr modifier..0x0000<br>Aug 25 17:28:03 woprjr0 Mkey...........0x0<br>Aug 25 17:28:03 woprjr0 DR SLID........0xffff<br>Aug 25 17:28:03 woprjr0 DR DLID........0xffff<br>
Aug 25 17:28:03 woprjr0 Madeye:sent SMP<br>Aug 25 17:28:03 woprjr0 MAD version....0x1<br>Aug 25 17:28:03 woprjr0 Class..........0x81 (Directed route SMP)<br>Aug 25 17:28:03 woprjr0 Class version..0x1<br>Aug 25 17:28:03 woprjr0 Method.........0x1 (Get)<br>
Aug 25 17:28:03 woprjr0 Status.........0x00<br>Aug 25 17:28:03 woprjr0 Hop pointer....0x0<br>Aug 25 17:28:03 woprjr0 Hop counter....0x0<br>Aug 25 17:28:03 woprjr0 Trans ID.......0x1b9d2a0f0cb7<br>Aug 25 17:28:03 woprjr0 Attr ID........0x15 (port info)<br>
Aug 25 17:28:03 woprjr0 Attr modifier..0x0001<br>Aug 25 17:28:03 woprjr0 Mkey...........0x0<br>Aug 25 17:28:03 woprjr0 DR SLID........0x02<br>Aug 25 17:28:03 woprjr0 DR DLID........0xffff<br><br>No response is shown for trid 0x1b9d2a0f0cb7...<br>
<br>As an aside I see the hop pointer is set to 1 at a lower level since<br>mad_encode does not do it.</blockquote>
<div> </div>
<div>Right; the SMI would do that.</div><br>
<blockquote class="gmail_quote" style="PADDING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT: #ccc 1px solid">So I guess the proper case for C14-9 would be "3) If Hop Pointer is equal to<br>Hop Count". (They are both 0.)</blockquote>
<div> </div>
<div>I'm not sure; maybe C14-9 4)</div><br>
<blockquote class="gmail_quote" style="PADDING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT: #ccc 1px solid">><br>> If so, what's the initial path at this point (or more specifically index 1<br>> of the initial path) ? I think that needs to be port 0 (if a switch) but<br>
> this is a little weird as I would think it should be handed to the SMA which<br>> is different cases in the spec.<br><br>Yes I think I was wrong on the case. But still wouldn't the SMI detect that<br>this is the end of the DRPath and simply hand it to the SMA.</blockquote>
<div> </div>
<div>Yes, that's what should happen.</div>
<div> </div><br>
<blockquote class="gmail_quote" style="PADDING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT: #ccc 1px solid">><br>><br>> > Then after processing<br>><br>><br>> by the SMA and doing the required returning initialization<br>
><br>> the SMI should return the packet as specified in C14-13<br>> > item 3 on line 9 page 812.<br>><br>><br>> I'm not sure it would use this case in the case of an empty DR pafh on<br>> return.<br>
<br>Actually I think it will use this. C14-9 item 3) states "the Hop Pointer<br>shall be incremented by 1" Therefore when the response is handed back to the<br>SMI the Hop pointer will be 1 and the hop count 0. And the SMI uses the<br>
DRSLID to send the packet back to the requester.</blockquote>
<div> </div>
<div>It goes up to the SMA and then when the response is to be made it goes through returning SMI initialization and handling.</div>
<div> </div>
<div>-- Hal</div><br>
<blockquote class="gmail_quote" style="PADDING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT: #ccc 1px solid">><br>> Am I wrong? In the end it does not matter as I have to make the software<br>> > work<br>
> > for all the hardware I have; so I will change the software.<br>><br>><br>> IMO it does matter as to where the problem lies (SMI or otherwise) and how<br>> the layers are comprised in the implementation.<br>
<br>Agreed. I am mainly confused because I have 2 different implementations of<br>this. My "old" switches seem to handle this case just fine. My "new"<br>switches do not. So I am really wondering what is going on.<br>
<br>Here is the above output for the same query which works with an "old" switch.<br><br>17:28:04 > ./smpquery -e -c portinfo 7 0 1<br>...<br>trid 1a4329de; HopCount 0; HopPointer 0; slid 2; dlid 65535; 0, drpath->cnt 0<br>
...<br><br>Aug 25 17:46:40 woprjr0 Madeye:sent SMP<br>Aug 25 17:46:40 woprjr0 MAD version....0x1<br>Aug 25 17:46:40 woprjr0 Class..........0x81 (Directed route SMP)<br>Aug 25 17:46:40 woprjr0 Class version..0x1<br>Aug 25 17:46:40 woprjr0 Method.........0x1 (Get)<br>
Aug 25 17:46:40 woprjr0 Status.........0x00<br>Aug 25 17:46:40 woprjr0 Hop pointer....0x0<br>Aug 25 17:46:40 woprjr0 Hop counter....0x0<br>Aug 25 17:46:40 woprjr0 Trans ID.......0x1ba01a4329de<br>Aug 25 17:46:40 woprjr0 Attr ID........0x15 (port info)<br>
Aug 25 17:46:40 woprjr0 Attr modifier..0x0001<br>Aug 25 17:46:40 woprjr0 Mkey...........0x0<br>Aug 25 17:46:40 woprjr0 DR SLID........0x02<br>Aug 25 17:46:40 woprjr0 DR DLID........0xffff<br>Aug 25 17:46:40 woprjr0 Madeye:recv SMP<br>
Aug 25 17:46:40 woprjr0 MAD version....0x1<br>Aug 25 17:46:40 woprjr0 Class..........0x81 (Directed route SMP)<br>Aug 25 17:46:40 woprjr0 Class version..0x1<br>Aug 25 17:46:40 woprjr0 Method.........0x81 (Get response)<br>
Aug 25 17:46:40 woprjr0 Status.........0x8000<br>Aug 25 17:46:40 woprjr0 Hop pointer....0x0<br>Aug 25 17:46:40 woprjr0 Hop counter....0x0<br>Aug 25 17:46:40 woprjr0 Trans ID.......0x1ba01a4329de<br>Aug 25 17:46:40 woprjr0 Attr ID........0x15 (port info)<br>
Aug 25 17:46:40 woprjr0 Attr modifier..0x0001<br>Aug 25 17:46:40 woprjr0 Mkey...........0x0<br>Aug 25 17:46:40 woprjr0 DR SLID........0x02<br>Aug 25 17:46:40 woprjr0 DR DLID........0xffff<br><br>Hop Pointer and Count are both 0 and things work just fine...<br>
<br>><br>> However, I wonder<br>> > where exactly the spec falls on this, because I think it will influence<br>> > where<br>> > the fix resides. If the spec does not allow this then I think it is fine<br>
> > to<br>> > have libibmad return an error since the user specified an invalid combined<br>> > DR<br>> > path. However, if this should be legal I think libibmad should work around<br>> > the bad hardware out there.<br>
><br>><br>> Is it hardware or firmware that needs fixing ? I think it may depend on the<br>> specific workaround for this as to whether it is acceptable as it might harm<br>> something else or might violate the spec.<br>
<br>I agree, however, if the switch hardware needs fixing I fear it is too late<br>for the ones I have. Firmware might be upgradable although I have had issues<br>with un-managed switches in the past.<br><br>So where do we put the fix in software?<br>
</blockquote>
<blockquote class="gmail_quote" style="PADDING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT: #ccc 1px solid">Ira<br><br>> -- Hal<br>><br>><br>> Thoughts?<br>> > Ira<br>> ><br>> > --<br>
> > Ira Weiny<br>> > Math Programmer/Computer Scientist<br>> > Lawrence Livermore National Lab<br>> > 925-423-8008<br>> > <a onclick="return top.js.OpenExtLink(window,event,this)" href="mailto:weiny2@llnl.gov" target="_blank">weiny2@llnl.gov</a><br>
> > _______________________________________________<br>> > general mailing list<br>> > <a onclick="return top.js.OpenExtLink(window,event,this)" href="mailto:general@lists.openfabrics.org" target="_blank">general@lists.openfabrics.org</a><br>
> > http://*<a onclick="return top.js.OpenExtLink(window,event,this)" href="http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general" target="_blank">lists.openfabrics.org/cgi-bin/mailman/listinfo/general</a><br>
> ><br>> > To unsubscribe, please visit<br>> > http://*<a onclick="return top.js.OpenExtLink(window,event,this)" href="http://openib.org/mailman/listinfo/openib-general" target="_blank">openib.org/mailman/listinfo/openib-general</a><br>
> ><br>><br><br><br>--<br>Ira Weiny<br>Math Programmer/Computer Scientist<br>Lawrence Livermore National Lab<br>925-423-8008<br><a onclick="return top.js.OpenExtLink(window,event,this)" href="mailto:weiny2@llnl.gov" target="_blank">weiny2@llnl.gov</a><br>
</blockquote></div><br>