[Openib-windows] Using of fast mutexes in WinIb

[Fab Tillier]

Hi Leo,

> From: Leonid Keller [mailto:leonid at mellanox.co.il]
> Sent: Tuesday, November 15, 2005 7:32 AM
> 
> Hi Fab,
> I come across the following problem: implementation of
> cl_mutex_acquire() via Fast Mutexes causes all the code in critical
> section to work at APC_LEVEL.
> 
> The first case i saw, was at the start-up of IpoIb driver, which takes
> mutex in __ipoib_pnp_cb and makes all MTHCA driver control verbs to work
> at APC_LEVEL, which is troublesome.
> (e.g., create_cq calls AllocateCommonBuffer, which requires
> PASSIVE_LEVEL).

Why does create_cq call AllocateCommonBuffer?  Are you making multiple
page-sized calls instead of a single larger call for the cases where the memory
required spans multiple pages?  Physically contiguous memory is a scarce
resource, so any time you can break up your requests into page sized requests
the better.

Is this for the CQE ring buffer?  Why not use regular memory?  The memory has to
be registered with the HCA anyway, right?

> There are several ways to solve the problem:
> 	1) Implement cl_mutex_acquire() via regular mutexes. It is a
> general and secure, but a bit ineffective way.
> 	2) Add to complib "regular mutexes" and use them in that
> function. It's more effective, but then we need  to check/fix all other
> uses of fast mutexes.

I don't think we should add a new mutex abstraction to complib that only has
meaning in the Windows kernel.  I'd much rather see the code just make native NT
calls for regular mutexes.

> 	3) Invent some other sync mechanism for that function. It lefts
> the problem with fast mutexes in other functions.
> 
> What do you think ?

Can we change these on a case-by-case basis?  That is, change the mutex in
IPoIB's PnP callback to a regular mutex for now, and change any others as we go
forward?

Finding incorrect IRQL issues should be quick if your calls assert that IRQL is
correct.

BTW, have you had a chance to look over the new verb API that would allow calls
at IRQL <= DISPATCH_LEVEL?  Any comments?  I'd really like to get past these
IRQL limitations.

- Fab