[ofa-general] Re: [RFC] [PATCH 0/3] 2.6.22 or 23 ib: add path record cache

[Michael S. Tsirkin]

> Quoting Sean Hefty <sean.hefty at intel.com>:
> Subject: RE: [RFC] [PATCH 0/3] 2.6.22 or 23 ib: add path record cache
> 
> >A straight-forward approach would be to listen for port up/down events
> >rather than or in addition to GID in/out, and do network discovery by DR SMPs.
> 
> I'm not entirely following you.  How would you listen for port up/down events?

Isn't there a way to get notice for this?

> And are you suggesting that all nodes do network discovery using DR SMPs?

I haven't thought this through yet. Basically, I just note that
caching the path until GID goes out of service isn't right -
since path parameters such as MTU or rate might change
without GID going out of service.

So what to do?

We could use DR SMPs to do network discovery and at least check
that paths are valid - it's not too much code (ibnetdiscover is just 800 lines)
and in a sense, that's actually putting an *SA* (not just cache) in each node.
Combined with GID IN/OUT notices we could get away from querying path records
completely.

Alternatively, we could notice that port state changed, and I think
we can figure out, from that, paths to which GIDs are affected, so that we can get
these anew from the SA.

> > Hmm. IPoIB by design does not handle heterogenious
> > networks too well (consider problems we have selecting bcast group rate).
> > MTU is also basically forced to 2K.
> > So this might not have been such a great example :)
> 
> I agree; I just didn't want to create something that was worse than what we have
> now.
> 
> >That's not entirely correct. For example, arp cache might get cleaned
> >by a timer, or by direct user request, or by other means.
> >When this happens, address handles and path records get freed.
> >Various IB events also trigger cache flush, such as the reregister event.
> 
> I'm not overly familiar with the code.  Is that what ipoib_neigh_destructor ends
> up doing?  (I need to walk through this to see where the path is freed.)

I re-checked and you are right - we keep the path record even
after all neighbours using it are gone. So of course this has
the same problem in that respect. At least the reregister/LID set
events flush the cache.


-- 
MST