[ofw] RE: Ibbus as a filter driver for mthca.

[Leonid Keller]

One important question was missed: How IBBUS coexist with WinVerbs
filter driver ? 
Today WinVerbs is a simple filter driver AFAIK and will sit under IBBUS.
How will it use IBAL interface (for CM, for example) ?
I'd say, that it should be a class filter driver, dependent on IBBUS -
if it is possible to require - and sit above IBBUS.
Sean ?


> -----Original Message-----
> From: Smith, Stan [mailto:stan.smith at intel.com] 
> Sent: Tuesday, August 05, 2008 9:11 PM
> To: Reuven Amitai; Leonid Keller; Tzachi Dar; Fab Tillier; 
> Alex Estrin; James Yang; Ishai Rabinovitz
> Cc: ofw at lists.openfabrics.org
> Subject: RE: Ibbus as a filter driver for mthca.
> 
> Hello,
>   I CC'ed the list as I forgot to when writing the original note.
> 
> Reuven Amitai wrote:
> > Hi Stan,
> >
> > I installed the driver (ibbus driver as a upper filter 
> driver) and ran 
> > basic tests (IOU will come soon).
> > Install/Uninstall using msi, Install using .inf, IPoIB 
> enable/disable 
> > with multiple HCAs, all seem to work for me.
> 
> Good news, thank you for investing time to test.
> 
> >
> > I have few questions from what I saw:
> > 1. Disable/Enable require restart:
> >     Trying to disable/enable mthca device require a reboot 
> in order to 
> > complete the operation.
> 
> Yes - mthca disable invokes a reboot request. Is this 'new' 
> behavior in that the 'current' mthca driver upon right-click 
> disable does not require a reboot?
> I'm not sure if this can be stopped/corrected?
> 
> Fab, is a reboot required or is it possible to defeat the 
> reboot by some .inf driver setting?
> 
> IPoIB enable/disable does not require a reboot.
> 
> >     Uninstall the device and re-install it using the .inf require 
> > reboot as well.
> >     Did you have the same phenomenon ? (It happens even right after 
> > clean install)
> 
> I would expect a device uninstall to request a reboot.
> In the bigger picture, customers not so often enable/disable 
> an HCA device. For you and I during development, we do; a 
> minor although irritating problem.
> 
> > 2. Attached DevTree images after clean install on a machine 
> with one 
> > HCA (mt25204) which has one port.
> >    From the images it seems that ibbus reports on the same 
> port twice 
> > and mthca also report for this port.
> >    (When tested on machine with 2 HCA with 2 port each, 8 child 
> > objects were reported by ibbus and 4 by mlx4_hca)
> >    I think that mthca report is happen since get_relation 
> function is 
> > still in p_ci_interface and mthca replies
> >    to PnP messages using this function (which goes to ibbus 
> > get_relation). I don't have a clue about ibbus reports.
> 
> Interesting, there does seem to be duplication in reporting relations.
> As I remember, mthca uses the p_ci_interface->get_relations 
> call only when mthca receives a QUERY_REMOVE_DEVICE; removal 
> relations reporting. Since the bus driver is correctly 
> reporting bus relations and there exists an implicit relation 
> between the bus driver and mthca driver (same device stack, 
> mthca cannot unload until it's instance of ibbus unloads). 
> Possibly, the mthca driver should not be reporting relations 
> like it does; one step closer to simple?  Do you know where 
> in mthca relations are reported?
> 
> > 3. Are Port/IPoIB devices related only to ibbus (and the 
> HCA driver is 
> > unaware of them) ?
> 
> The mthca driver registers with the bus driver via the CI 
> interface. The bus driver has registered for IBAL PNP events, 
> which PORT_ADD/REMOVE and IOU_ADD/REMOVE are explicitly 
> handled by ibbus.
> The HCA driver is unaware of IPoIB and/or IOU devices and 
> aware of port devices.
> The CI->get_relations call from the HCA driver informs the 
> HCA about existing bus relations and the implied dependence 
> on an HCA port.
> 
> >     Did you change HCA driver ? (Ignore PnP messages, change HCA 
> > registration to ibbus to ibbus query for HCA driver interface)
> 
> Mthca driver was not modified, so far, for any of the bus filter work.
> 
> >     Could you describe what are the changes that you have 
> done in the 
> > bus driver ?
> >     (In which sections the ibbus changed ?)
> 
> The big change for ibbus is the recognition and support for 
> multiple HCAs; support for multiple calls to 
> bus_add_device(). A Bus Filter Instance (BFI) is allocated 
> for each HCA. This BFI instance exists as a bus_filter_t* in 
> a vector of bus_filters; yes there is a max(16) number of 
> HCAs support per system. The max is an implementation 
> artifact, a linked list would remove the max although systems 
> tend to not have many HCAs so the list felt like overkill at 
> this time; a possible stepwise refinement once larger issues 
> are resolved.
> The routine 'bus_add_device()' is called for each HCA 
> discovered via PNP. Add_device() allocates a BFI slot for the 
> new HCA device; HCA will register with IBAL via the CI 
> interface. When the 1st PORT/IOU ADD PNP event occurs, a 
> port/iou manager is created for the specific HCA; just like 
> the current ibbus, only real difference is ibbus is aware of 
> multiple HCAs and managers.
> The BFI enables the IBAL PNP callback mechanism to match the 
> PNP event to an HCA and locate the port and/or iou manger for 
> the HCA in order to handle the PNP event.
> The PNP record 'context' is now a pointer to a 
> port/iou_context block which contains a BFI pointer and 
> current port/iou object pointer as it does in the current ibbus.
> 
> The other big change is the bus driver does not report 
> removal relations and now reports bus relations; this may be 
> part of the double reporting problem.
> 
> Major mods to bus_port_mgr.c, bus_iou_mgr.c and bus_pnp.c; 
> see .patches file in branches\winverbs\core\bus\kernel.
> 
> > 4. What is the difference of having ibbus as a class filter driver 
> > instead of regular filter driver ?
> >     How this will integrate with WinVerbs class filter driver ?
> 
> An HCA driver is a class driver (InfiniBandHca, although Svr 
> 2008 already pre-defines the InfiniBandController class). 
> Ibbus is an upper class filter driver and expects HCAs to be 
> of the same class, hence the Bus Filter Instance concept.
> Bottom line is ibbus removes the IbInstaller.dll co-installer 
> use and gets ibbus/ibal and the HCA all in the same device 
> stack thus stepping towards a less complex (no co-installer) 
> IB stack. Longer term simpler could be defined as migrating 
> driver PNP handling out of IBAL and back into Windows where 
> it belongs, thus making IBBUS/IBAL much simpler in the long 
> run (think kmdf).
> Additionally, ibbus as a filter driver becomes much more 2008 
> installer friendly as ibbus/ibal/complib/hca-driver collapse 
> into a single IBCORE .inf file with IOU concerns partitioned 
> into a separate .inf file (ib_iou.inf).
> 
> Good questions.
> 
> Thanks,
> 
> Stan.
> 
> 
> >
> > Thanks, Reuven.
> >
> > -----Original Message-----
> > From: Smith, Stan [mailto:stan.smith at intel.com]
> > Sent: Saturday, August 02, 2008 4:45 AM
> > To: Leonid Keller; Tzachi Dar; Fab Tillier; Alex Estrin; 
> James Yang; 
> > Reuven Amitai; Ishai Rabinovitz
> > Cc: Woodruff, Robert J
> > Subject: Ibbus as a filter driver for mthca.
> >
> > Gentlemen,
> >   The ibbus driver as a upper filter driver is showing 
> signs of life.
> > As a filter driver, no IB coinstaller is used making it 
> more Svr 2008 
> > install friendly.
> > The big change is supporting multiple HCAs, hence multiple
> > bus_add_device() calls along with separate port and iou 
> managers per 
> > HCA.
> >
> > For mthca devices (single and multiple HCAs per system) the ibbus 
> > filter driver is functioning correctly for the following areas:
> >
> >   WinOF installer load/unload via devcon.exe commands; need to try 
> > dpinst.exe ASAP.
> >
> >   Found New Hardware wizard install via a modified mthca.inf.
> >
> >   IPoIB enable/disable working correctly with multiple HCAs.
> >
> >   Disable with following uninstall.
> >
> >   Clean IB stack disable and uninstall.
> >
> >   Clean system boot and shutdown - no lingering object reference 
> > against the HCA.
> >
> > The reason for this note is to solicit help in debugging IOU 
> > functionality as I have no IOUs to test; additionally a wider debug 
> > exposure is highly desirable.
> >
> > The source, with moderate debug enabled in a checked build, is 
> > available from \gen1\branches\winverbs\core\bus\kernel. The kernel 
> > folder is a direct replacement for trunk\core\bus\kernel.
> > You must use the modified mthca.inf file to load as it's a blend of 
> > the current mthca.inf plus portions of ib_bus.inf. Only 
> mthca.inf is 
> > required to load the core IB stack. The modified mthca is in 
> > gen1\branches\winverbs\core\bus\kernel\etc for now.
> >
> > I have been testing using mthca devices on svn.1433, will 
> be testing 
> > with the head-of-svn + Leonid's email patch come Monday; 
> Sean H. tells 
> > me the head + Leonid's patch works fine for now.
> >
> > ConnectX has not been tested yet. Will be starting ConnectX 
> soon using 
> > the mlx4_hca2.inf file which Leonid kindly sent me.
> >
> > If some of you could kindly check build and test this filter driver 
> > with your IOUs I would be most thankful and most willing to 
> assist in 
> > any way I can. Code review is most welcome as I'm sure I 
> have missed 
> > issues along the way.
> >
> > Thank you all in advance!
> >
> > Stan.
> 
>