[Openib-windows] RE: open_ca

[Fab Tillier]

Copying the mailing list as this information might be of interest to others.

> From: Leonid Keller [mailto:leonid at mellanox.co.il]
> Sent: Sunday, September 25, 2005 10:06 AM
> 
> Hi Fab,
> Could you shortly explain the policy with open_ca ?
> I mean, VAPI 3.2 driver didn't expose open_ca API at all: it did it itself
> once in the life.
> WinIb driver exposes this API and calls it itself already in __hca_register.

The HCA driver calls it internally to get the CA GUIDs and what not.  It then
closes it and the HCA is re-opened in the call to ci_open_ca.

> Also all ULPs call this function.

No, ULPs call ib_open_ca, which doesn't turn into any call to the HCA driver at
all.  For user-mode clients, an additional call to the HCA driver is made -
ci_um_open_ca - to establish a user context with the HCA and allow exchanging
parameters with the UVP.

> Let's say, we have only one HCA.
> My questions are:
>     Who owns CA object, created on the first, driver's call to open_ca ?

IBAL owns the CA object - it is represented by the CI_CA object.

>     How (=by whom) it is used ?

Every client that calls ib_open_ca ends up referencing it internally to IBAL.

>     What happens on ULP's call to open_ca: are new CA objects created ?

No, the HCA driver is unaware of clients.  All client management is done in
IBAL.

>     Does mlnx_open_ca get called more than once ?

No, it doesn't.  The HCA driver detects this and fails subsequent calls.

>     What happens when an application calls to ib_open_ca ? Is user CA object
> somehow connected to kernel CA object ?

For a user-mode process, the ib_open_ca code path makes a call to ci_um_open_ca
which establishes a user-mode context for the HCA.  I added this to eliminate
internal IOCTL generation in the UVP.

I think the CR mapping issue highlights the need to change things somewhat.
IBAL was designed to minimize the amount of work done in the HCA driver,
especially with respect to user-management.  This model avoids duplicating user
management in each HCA driver.

Initially (before I added the ci_um_open_ca call), the model forced HCA drivers
to implement their own internal IOCTL paths or find alternate ways of exchanging
information between the UVP and KVP.  This resulted in a pretty ugly
architecture, as well as some shady hacks in UVPs (like using the query_ca call
to establish context with the KVP).  I made changes to allow the UVP to
establish a context with the KVP during an app's ib_open_ca call, which allowed
me to eliminate all internal IOCTL paths from the HCA driver (I like eliminating
code) and cleaning things up a bit.

I think we're at a point where it would make sense for the HCA driver to just
initialize the device internally, and allow multiple open_ca calls (basically,
have ci_um_open_ca replace ci_open_ca).  I will start working on this, but will
be traveling this week and this will slow me down.

- Fab