[ewg] Re: pkey change handling patch

[Michael S. Tsirkin]

> Quoting Roland Dreier <rdreier at cisco.com>:
> Subject: Re: pkey change handling patch
> 
>  > Could be a good idea, but note this will require creating another
>  > WQ for cache updates, otherwise e.g. IPoIB
>  > will deadlock waiting for it.
> 
> Actually, on further thought, it's kind of a stupid idea.  The whole
> point of the cache module is to be usable in places where blocking
> isn't allowed.  If it's being called from a context where we know we
> can block, then there's no point in going through the cache at all.
> 
>  > By the way, are there any users for the non-blocking API?
>  > Maybe we can simply relax the requirement, and make all API's
>  > blocking?
> 
> Well, at least mthca is using it when posting WQEs to MLX QPs.  But it
> could keep its own internal cache (which would be much simpler, and
> easier to keep in sync since it sees all MADs that change the cache as
> they go by).
> 
> If there are no other users for a non-blocking API then we could tear
> out the whole caching mess and be much happier.
> 
> At least IPoIB and SRP seem like they could live without the cache,
> with the addition of ib_find_pkey() and ib_find_gid() convenience
> functions, and they're the only users in infiniband/ulp.
> infiniband/core would take a little more auditing.

Good point.
So since all this thread was started by Moni because of IPoIB,
the path is clear in that respect, and would already be a step in the
right direction:

- a patch to add ib_find_pkey() and ib_find_gid() to core
- a patch to replace cache usage in IPoIB / SRP with uncached
  hardware accesses on top of this
- pkey change handling patch on top of these

Moni, what do you think?

As a follow-up, we can then
- get rid of cache in mthca by implementing device-specific cache.
- audit core for cache usage - most likely we'll find everything can be
  done from a context where we can block

By the way, here's yet another bright idea:

I *think* we can keep this provider-specific cache updated by snooping incoming
MADs in driver.  And if it can be done this way, in all providers, we might be able to
simply require that query_pkey/query_gid in providers do not sleep.
If that's true, we'll save ourselves the work of auditing core - ib_find_pkey()
and ib_find_gid() will be a drop-in replacement for cache.

Roland, what do you think?

-- 
MST