[ofw] RE: Ibbus as a filter driver for mthca.

[Smith, Stan]

Hello,
  I CC'ed the list as I forgot to when writing the original note.

Reuven Amitai wrote:
> Hi Stan,
>
> I installed the driver (ibbus driver as a upper filter driver) and ran
> basic tests (IOU will come soon).
> Install/Uninstall using msi, Install using .inf, IPoIB enable/disable
> with multiple HCAs, all seem to work for me.

Good news, thank you for investing time to test.

>
> I have few questions from what I saw:
> 1. Disable/Enable require restart:
>     Trying to disable/enable mthca device require a reboot in order to
> complete the operation.

Yes - mthca disable invokes a reboot request. Is this 'new' behavior in that the 'current' mthca driver upon right-click disable does not require a reboot?
I'm not sure if this can be stopped/corrected?

Fab, is a reboot required or is it possible to defeat the reboot by some .inf driver setting?

IPoIB enable/disable does not require a reboot.

>     Uninstall the device and re-install it using the .inf require
> reboot as well.
>     Did you have the same phenomenon ? (It happens even right after
> clean install)

I would expect a device uninstall to request a reboot.
In the bigger picture, customers not so often enable/disable an HCA device. For you and I during development, we do; a minor although irritating problem.

> 2. Attached DevTree images after clean install on a machine with one
> HCA (mt25204) which has one port.
>    From the images it seems that ibbus reports on the same port twice
> and mthca also report for this port.
>    (When tested on machine with 2 HCA with 2 port each, 8 child
> objects were reported by ibbus and 4 by mlx4_hca)
>    I think that mthca report is happen since get_relation function is
> still in p_ci_interface and mthca replies
>    to PnP messages using this function (which goes to ibbus
> get_relation). I don't have a clue about ibbus reports.

Interesting, there does seem to be duplication in reporting relations.
As I remember, mthca uses the p_ci_interface->get_relations call only when mthca receives a QUERY_REMOVE_DEVICE; removal relations reporting. Since the bus driver is correctly reporting bus relations and there exists an implicit relation between the bus driver and mthca driver (same device stack, mthca cannot unload until it's instance of ibbus unloads). Possibly, the mthca driver should not be reporting relations like it does; one step closer to simple?  Do you know where in mthca relations are reported?

> 3. Are Port/IPoIB devices related only to ibbus (and the HCA driver is
> unaware of them) ?

The mthca driver registers with the bus driver via the CI interface. The bus driver has registered for IBAL PNP events, which PORT_ADD/REMOVE and IOU_ADD/REMOVE are explicitly handled by ibbus.
The HCA driver is unaware of IPoIB and/or IOU devices and aware of port devices.
The CI->get_relations call from the HCA driver informs the HCA about existing bus relations and the implied dependence on an HCA port.

>     Did you change HCA driver ? (Ignore PnP messages, change HCA
> registration to ibbus to ibbus query for HCA driver interface)

Mthca driver was not modified, so far, for any of the bus filter work.

>     Could you describe what are the changes that you have done in the
> bus driver ?
>     (In which sections the ibbus changed ?)

The big change for ibbus is the recognition and support for multiple HCAs; support for multiple calls to bus_add_device(). A Bus Filter Instance (BFI) is allocated for each HCA. This BFI instance exists as a bus_filter_t* in a vector of bus_filters; yes there is a max(16) number of HCAs support per system. The max is an implementation artifact, a linked list would remove the max although systems tend to not have many HCAs so the list felt like overkill at this time; a possible stepwise refinement once larger issues are resolved.
The routine 'bus_add_device()' is called for each HCA discovered via PNP. Add_device() allocates a BFI slot for the new HCA device; HCA will register with IBAL via the CI interface. When the 1st PORT/IOU ADD PNP event occurs, a port/iou manager is created for the specific HCA; just like the current ibbus, only real difference is ibbus is aware of multiple HCAs and managers.
The BFI enables the IBAL PNP callback mechanism to match the PNP event to an HCA and locate the port and/or iou manger for the HCA in order to handle the PNP event.
The PNP record 'context' is now a pointer to a port/iou_context block which contains a BFI pointer and current port/iou object pointer as it does in the current ibbus.

The other big change is the bus driver does not report removal relations and now reports bus relations; this may be part of the double reporting problem.

Major mods to bus_port_mgr.c, bus_iou_mgr.c and bus_pnp.c; see .patches file in branches\winverbs\core\bus\kernel.

> 4. What is the difference of having ibbus as a class filter driver
> instead of regular filter driver ?
>     How this will integrate with WinVerbs class filter driver ?

An HCA driver is a class driver (InfiniBandHca, although Svr 2008 already pre-defines the InfiniBandController class). Ibbus is an upper class filter driver and expects HCAs to be of the same class, hence the Bus Filter Instance concept.
Bottom line is ibbus removes the IbInstaller.dll co-installer use and gets ibbus/ibal and the HCA all in the same device stack thus stepping towards a less complex (no co-installer) IB stack. Longer term simpler could be defined as migrating driver PNP handling out of IBAL and back into Windows where it belongs, thus making IBBUS/IBAL much simpler in the long run (think kmdf).
Additionally, ibbus as a filter driver becomes much more 2008 installer friendly as ibbus/ibal/complib/hca-driver collapse into a single IBCORE .inf file with IOU concerns partitioned into a separate .inf file (ib_iou.inf).

Good questions.

Thanks,

Stan.


>
> Thanks, Reuven.
>
> -----Original Message-----
> From: Smith, Stan [mailto:stan.smith at intel.com]
> Sent: Saturday, August 02, 2008 4:45 AM
> To: Leonid Keller; Tzachi Dar; Fab Tillier; Alex Estrin; James Yang;
> Reuven Amitai; Ishai Rabinovitz
> Cc: Woodruff, Robert J
> Subject: Ibbus as a filter driver for mthca.
>
> Gentlemen,
>   The ibbus driver as a upper filter driver is showing signs of life.
> As a filter driver, no IB coinstaller is used making it more Svr 2008
> install friendly.
> The big change is supporting multiple HCAs, hence multiple
> bus_add_device() calls along with separate port and iou managers per
> HCA.
>
> For mthca devices (single and multiple HCAs per system) the ibbus
> filter driver is functioning correctly for the following areas:
>
>   WinOF installer load/unload via devcon.exe commands; need to try
> dpinst.exe ASAP.
>
>   Found New Hardware wizard install via a modified mthca.inf.
>
>   IPoIB enable/disable working correctly with multiple HCAs.
>
>   Disable with following uninstall.
>
>   Clean IB stack disable and uninstall.
>
>   Clean system boot and shutdown - no lingering object reference
> against the HCA.
>
> The reason for this note is to solicit help in debugging IOU
> functionality as I have no IOUs to test; additionally a wider debug
> exposure is highly desirable.
>
> The source, with moderate debug enabled in a checked build, is
> available from \gen1\branches\winverbs\core\bus\kernel. The kernel
> folder is a direct replacement for trunk\core\bus\kernel.
> You must use the modified mthca.inf file to load as it's a blend of
> the current mthca.inf plus portions of ib_bus.inf. Only mthca.inf is
> required to load the core IB stack. The modified mthca is in
> gen1\branches\winverbs\core\bus\kernel\etc for now.
>
> I have been testing using mthca devices on svn.1433, will be testing
> with the head-of-svn + Leonid's email patch come Monday; Sean H. tells
> me the head + Leonid's patch works fine for now.
>
> ConnectX has not been tested yet. Will be starting ConnectX soon using
> the mlx4_hca2.inf file which Leonid kindly sent me.
>
> If some of you could kindly check build and test this filter driver
> with your IOUs I would be most thankful and most willing to assist in
> any way I can. Code review is most welcome as I'm sure I have missed
> issues along the way.
>
> Thank you all in advance!
>
> Stan.