[openib-general] IPoIB and lid change

[Eitan Zahavi]

Hal Rosenstock wrote:
> Hi Eitan,
> 
> On Mon, 2006-02-13 at 10:23, Eitan Zahavi wrote:
> 
>>Hi,
>>
>>I had a long discussion today with Michael, Yael and Tziporet regarding
>>this issue.
>>We have got to the following conclusions/proposal:
>>
>>1. As we use only GID[0] (that can not change) and a QP that is reserved
>>for the interface even if it is down we actually "never" change IPoIB
>>MAC (unless you download the module).
> 
> 
> remove the module
Sure - pardon my English.
> 
> 
>>So unsolicited ARP reply is not a must. The MAC (which is GID,QP in IPoIB) 
>>is kept even if the LID changes.
>>
>>2. When a new SM is brought up it optionally can send
>>ClientReRegistration which should cause the entire UD AV cache (and the
>>new SA Path Cache) to be flushed.
> 
> 
> Would this be an SM requirement ? 
This is an optional SM feature. So I can not "require" it.
Implementation of the "UnPath" event would suffice but will probably be "optional" too.
The entire UD AV cache includes both
> unicast and multicast, right ?
Yes both require to be flushed.

> 
> 
>>3. When subnets are merged there is currently no way to tell if there
>>were any LID changes. The SM does not need to do any "Set(PortInfo)" on
>>a node if no change is required so the remote node (that did not change
>>LID) will not know about any such change. There are several solutions
>>for this problem. The one we believe should be promoted is the concept
>>of "UnPath" Notice:
> 
> 
> UnPath will be needed for QoS too. It has been discussed in that context
> as well.
Agree.
> 
> 
>>a. In IB any class manager (SM) can generate Traps carrying a Notice
>>attribute. We  
>>    propose a new Notice of trap number = YYY which will mean UnPath
>>message.
>>    The notice will carry several fields of the path record as its
>>DataDetails field and also 
>>    a component mask. (the DataDetails field is 432 bits long).
>>    So the following fields are to be included in the DataDetails:
>>    SLID, DLID, P_KEY, TCLASS, SL, compMask.
>>    The component mask will allow to wildcard the fields values such
>>that:
>>    * an UnPath Notice including a component mask = 0 will UnPath all
>>paths.
>>    * an UnPath Notice including a component mask = 1 will UnPath all
>>paths from the   
>>      LID = the SLID included in the Notice.
>>
>>b. The IPoIB will have to register with the SA to receive these notices.
>>
>>c. The SM needs to be smart with the way the notices are being built:
>>    The SM should be able to coalesce multiple change events to one
>>notice by using the 
>>    wildcard (zero compmask) as appropriate for the case inspected by
>>the SM.
>>
>>With this kind of coalescing the SM does not need to generate O(N^2)
>>Reports but only O(N) Reports. As you might guess any discovery and
>>setting of the fabric require O(N^2/32) just for LFTs setting
> 
> 
> Or perhaps these are distributed on some MC group reserved for this
> (unpath) ?
I would prefer to pay the price of sending O(N) packets and "know" they arrive.
Any node not responding to the Report with ReportRepress will get some "retries"
and if even that does not help we at least know about it. Using MC will
not provide this knowledge - even though we could use a scheme where we "retry" each
send multiple times. I consider this a secondary issue. I am glad we agree on the
overall concept.
> 
> 
>>s so our
>>problem of distributing these N reports is not so big.
>>
>> My plan is to bring this UnPath Notice to the IBTA MgtWG for
>>discussion. 
> 
> 
> Sounds good; (That's where this will need to be standardized.)
Just wanted to see if this will make sense on the OpenIB list first.
> 
> -- Hal
> 
> 
>>Thanks
>>
>>Eitan
>>
> 
>