<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=US-ASCII">
<META NAME="Generator" CONTENT="MS Exchange Server version 5.5.2654.45">
<TITLE>RE: [Openib-windows] [BUG] Errors with multiple HCAs in a single system</TITLE>
</HEAD>
<BODY>
<P><FONT SIZE=2>SB</FONT>
</P>
<P><FONT SIZE=2>> -----Original Message-----</FONT>
<BR><FONT SIZE=2>> From: Fab Tillier [<A HREF="mailto:ftillier@silverstorm.com">mailto:ftillier@silverstorm.com</A>]</FONT>
<BR><FONT SIZE=2>> Sent: Monday, October 17, 2005 9:16 PM</FONT>
<BR><FONT SIZE=2>> To: openib-windows@openib.org</FONT>
<BR><FONT SIZE=2>> Subject: [Openib-windows] [BUG] Errors with multiple HCAs in a single</FONT>
<BR><FONT SIZE=2>> system</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> I am trying to run both a PCI-X and PCI-Express HCA in a </FONT>
<BR><FONT SIZE=2>> single server </FONT>
</P>
<P><FONT SIZE=2>I guess, you are the first one, that's trying that ... :(</FONT>
</P>
<P><FONT SIZE=2>> to</FONT>
<BR><FONT SIZE=2>> compare performance, and cannot enable both devices </FONT>
<BR><FONT SIZE=2>> simultaneously. Which ever</FONT>
<BR><FONT SIZE=2>> HCA is starts second fails with the following output to the debugger:</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> <1> MDT(1): prepare_intr_resources: Failed to MOSAL_ISR_set </FONT>
<BR><FONT SIZE=2>> MOSAL ret=-1</FONT>
<BR><FONT SIZE=2>> <1> MDT(4): THH_eventp_create: Cannot set interrupt resources.</FONT>
<BR><FONT SIZE=2>> <1> MDT(1): THH_hob_open_hca: could not create eventp (-255)</FONT>
<BR><FONT SIZE=2>> <1> MDT(1): THH_hob_close_hca: Device already closed</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> It looks like the device lookup in MOSAL_ISR_set uses a generic name</FONT>
<BR><FONT SIZE=2>> "InfiniHost" and thus matches the first entry in the table.</FONT>
</P>
<P><FONT SIZE=2>Yes, a bug in prepare_intr_resources(), which calls MOSAL_ISR_set() with "InfiniHost" built_in parameter instead of real name. Try to change it by eventp->hob.dev_name.</FONT></P>
<P><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> Why is this doing a lookup? Doesn't the code know exactly </FONT>
<BR><FONT SIZE=2>> which device is being</FONT>
<BR><FONT SIZE=2>> initialized? Why put the device in the MOSAL_dev_db and then </FONT>
<BR><FONT SIZE=2>> search for it?</FONT>
</P>
<P><FONT SIZE=2>MOSAL was born in Linux as a separate module, that performs all OS-dependent things.</FONT>
<BR><FONT SIZE=2>It is to do that on base of the function parameters.</FONT>
<BR><FONT SIZE=2>When i start to port MOSAL to Windows, i saw that in some cases (like with MOSAL_ISR_set()) MOSAL can't do that, because it needs some information, found only in the posession of the driver.</FONT></P>
<P><FONT SIZE=2>That's why i added MOSAL DB, which duplicates some Driver device information.</FONT>
</P>
<P><FONT SIZE=2>There is no MOSAL in new driver, so there will be no that problem ...</FONT>
</P>
<BR>
<P><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> I set a breakpoint and manually helped the code find the </FONT>
<BR><FONT SIZE=2>> right entry, and things</FONT>
<BR><FONT SIZE=2>> moved forward from here but quickly failed again when </FONT>
<BR><FONT SIZE=2>> creating a special QP:</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> ~1:mlnx_query_qp() !ERROR!: completes with ERROR status </FONT>
<BR><FONT SIZE=2>> IB_INVALID_QP_HANDLE</FONT>
<BR><FONT SIZE=2>> ~1:mlnx_create_spl_qp() !ERROR!: completes with ERROR status</FONT>
<BR><FONT SIZE=2>> IB_INVALID_QP_HANDLE</FONT>
<BR><FONT SIZE=2>> ~1:al:create_spl_qp_svc() !ERROR!: ib_get_spl_qp failed, </FONT>
<BR><FONT SIZE=2>> IB_INVALID_QP_HANDLE</FONT>
<BR><FONT SIZE=2>> ~1:al:create_spl_qp_svc() ]</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> Now, the fact that create_spl_qp can return </FONT>
<BR><FONT SIZE=2>> IB_INVALID_QP_HANDLE seems flawed to</FONT>
<BR><FONT SIZE=2>> me. To make things worse, the HCA driver doesn't clean up the QP it</FONT>
<BR><FONT SIZE=2>> successfully created internally, and that resource is thus leaked.</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> Apparently, the same issue happens with normal QP creation, </FONT>
<BR><FONT SIZE=2>> judging by the</FONT>
<BR><FONT SIZE=2>> output from IPoIB trying to load:</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> ~0:mlnx_query_qp() !ERROR!: completes with ERROR status </FONT>
<BR><FONT SIZE=2>> IB_INVALID_QP_HANDLE</FONT>
<BR><FONT SIZE=2>> ~0:mlnx_create_qp() !ERROR!: completes with ERROR status </FONT>
<BR><FONT SIZE=2>> IB_INVALID_QP_HANDLE</FONT>
<BR><FONT SIZE=2>> ~0:__ib_mgr_init() !ERROR!: ib_create_qp returned IB_INVALID_QP_HANDLE</FONT>
<BR><FONT SIZE=2>> ~0:__ib_mgr_init() ]</FONT>
<BR><FONT SIZE=2>> ~0:__port_init() !ERROR!: __ib_mgr_init returned IB_INVALID_QP_HANDLE</FONT>
<BR><FONT SIZE=2>> ~0:__port_init() ]</FONT>
<BR><FONT SIZE=2>> ~0:ipoib_create_port() !ERROR!: ipoib_port_init returned </FONT>
<BR><FONT SIZE=2>> IB_INVALID_QP_HANDLE.</FONT>
<BR><FONT SIZE=2>> ~0:ipoib_create_port() ]</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> IPoIB then tries to tear things down and the code asserts </FONT>
<BR><FONT SIZE=2>> because PD destruction</FONT>
<BR><FONT SIZE=2>> failed:</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> ~0:mlnx_deallocate_pd() !ERROR!: completes with ERROR status </FONT>
<BR><FONT SIZE=2>> IB_RESOURCE_BUSY</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> The resource is busy due to the leak of the QP when </FONT>
<BR><FONT SIZE=2>> mlnx_create_qp failed.</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> - Fab</FONT>
<BR><FONT SIZE=2>> </FONT>
<BR><FONT SIZE=2>> _______________________________________________</FONT>
<BR><FONT SIZE=2>> openib-windows mailing list</FONT>
<BR><FONT SIZE=2>> openib-windows@openib.org</FONT>
<BR><FONT SIZE=2>> <A HREF="http://openib.org/mailman/listinfo/openib-windows" TARGET="_blank">http://openib.org/mailman/listinfo/openib-windows</A></FONT>
<BR><FONT SIZE=2>> </FONT>
</P>
</BODY>
</HTML>