[openib-general] IPoIB and lid change

[Eitan Zahavi]

Hi,

I had a long discussion today with Michael, Yael and Tziporet regarding
this issue.
We have got to the following conclusions/proposal:

1. As we use only GID[0] (that can not change) and a QP that is reserved
for the interface even if it is down we actually "never" change IPoIB
MAC (unless you download the module). So unsolicited ARP reply is not a
must. The MAC (which is GID,QP in IPoIB) is kept even if the LID
changes.

2. When a new SM is brought up it optionally can send
ClientReRegistration which should cause the entire UD AV cache (and the
new SA Path Cache) to be flushed.

3. When subnets are merged there is currently no way to tell if there
were any LID changes. The SM does not need to do any "Set(PortInfo)" on
a node if no change is required so the remote node (that did not change
LID) will not know about any such change. There are several solutions
for this problem. The one we believe should be promoted is the concept
of "UnPath" Notice:

a. In IB any class manager (SM) can generate Traps carrying a Notice
attribute. We  
    propose a new Notice of trap number = YYY which will mean UnPath
message.
    The notice will carry several fields of the path record as its
DataDetails field and also 
    a component mask. (the DataDetails field is 432 bits long).
    So the following fields are to be included in the DataDetails:
    SLID, DLID, P_KEY, TCLASS, SL, compMask.
    The component mask will allow to wildcard the fields values such
that:
    * an UnPath Notice including a component mask = 0 will UnPath all
paths.
    * an UnPath Notice including a component mask = 1 will UnPath all
paths from the   
      LID = the SLID included in the Notice.

b. The IPoIB will have to register with the SA to receive these notices.

c. The SM needs to be smart with the way the notices are being built:
    The SM should be able to coalesce multiple change events to one
notice by using the 
    wildcard (zero compmask) as appropriate for the case inspected by
the SM.

With this kind of coalescing the SM does not need to generate O(N^2)
Reports but only O(N) Reports. As you might guess any discovery and
setting of the fabric require O(N^2/32) just for LFTs settings so our
problem of distributing these N reports is not so big.

 My plan is to bring this UnPath Notice to the IBTA MgtWG for
discussion. 

Thanks

Eitan