<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=US-ASCII">
<META NAME="Generator" CONTENT="MS Exchange Server version 5.5.2654.45">
<TITLE>RE: geting remote and locale ip addresses</TITLE>
</HEAD>
<BODY>
<P><FONT SIZE=2>The IOCTL message will be used in the implementation (option #3).</FONT>
</P>
<P><FONT SIZE=2>I'll soon start a new thread about the questions that were raised by you.</FONT>
</P>
<P><FONT SIZE=2>Thanks</FONT>
<BR><FONT SIZE=2>Tzachi</FONT>
</P>
<P><FONT SIZE=2>>-----Original Message-----</FONT>
<BR><FONT SIZE=2>>From: Fab Tillier [<A HREF="mailto:ftillier@silverstorm.com">mailto:ftillier@silverstorm.com</A>]</FONT>
<BR><FONT SIZE=2>>Sent: Wednesday, September 07, 2005 12:49 AM</FONT>
<BR><FONT SIZE=2>>To: 'Tzachi Dar'; openib-windows@openib.org</FONT>
<BR><FONT SIZE=2>>Subject: RE: geting remote and locale ip addresses</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>>> From: Tzachi Dar [<A HREF="mailto:tzachid@mellanox.co.il">mailto:tzachid@mellanox.co.il</A>]</FONT>
<BR><FONT SIZE=2>>> Sent: Monday, September 05, 2005 3:11 PM</FONT>
<BR><FONT SIZE=2>>></FONT>
<BR><FONT SIZE=2>>> Hi Fab,</FONT>
<BR><FONT SIZE=2>>></FONT>
<BR><FONT SIZE=2>>> I have started looking at the implementation of this functionality in the</FONT>
<BR><FONT SIZE=2>>> IPOIB driver. It seems that the first thing that should be decided is the</FONT>
<BR><FONT SIZE=2>>> way that the IPOIB driver will expose some way to talk with other</FONT>
<BR><FONT SIZE=2>>components</FONT>
<BR><FONT SIZE=2>>> (also from the user mode).</FONT>
<BR><FONT SIZE=2>>></FONT>
<BR><FONT SIZE=2>>> To me it seems that there are 4 ways to achieve this task:</FONT>
<BR><FONT SIZE=2>>></FONT>
<BR><FONT SIZE=2>>> 1) It seems to me that "Device Interface Classes" as they are called</FONT>
<BR><FONT SIZE=2>>in</FONT>
<BR><FONT SIZE=2>>> the MSDN (search for this strings in MSDN in "titles only") are the way</FONT>
<BR><FONT SIZE=2>>that</FONT>
<BR><FONT SIZE=2>>> MSDN recommends to do the task. (Do not confuse device interfaces with</FONT>
<BR><FONT SIZE=2>>the</FONT>
<BR><FONT SIZE=2>>> interfaces that drivers can export in response to an</FONT>
<BR><FONT SIZE=2>>IRP_MN_QUERY_INTERFACE</FONT>
<BR><FONT SIZE=2>>> request. That IRP is used to pass routine entry points between kernel-</FONT>
<BR><FONT SIZE=2>>mode</FONT>
<BR><FONT SIZE=2>>> drivers.) On the other hand these functions don't seem to allow any way</FONT>
<BR><FONT SIZE=2>>of</FONT>
<BR><FONT SIZE=2>>> custom connection and are therefore (probably) can not be used.</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>>The IRP_MN_QUERY_INTERFACE method can be used to find a device that</FONT>
<BR><FONT SIZE=2>>supports a</FONT>
<BR><FONT SIZE=2>>particular interface using the SetupDiXxx functions. There are also</FONT>
<BR><FONT SIZE=2>>notification mechanisms to allow an application to release the device when</FONT>
<BR><FONT SIZE=2>>the</FONT>
<BR><FONT SIZE=2>>interface becomes unavailable. Once the device is resolved, an application</FONT>
<BR><FONT SIZE=2>>would send requests via IOCTLs.</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>>This isn't a bad method, but still requires the NDIS miniport to handle</FONT>
<BR><FONT SIZE=2>>IOCTLs.</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>>> 2) Use the IRP_MN_QUERY_INTERFACE way. It seems that this way is used</FONT>
<BR><FONT SIZE=2>>by</FONT>
<BR><FONT SIZE=2>>> our stack to pass interfaces from one component to another (for example</FONT>
<BR><FONT SIZE=2>>the</FONT>
<BR><FONT SIZE=2>>> GUID_IB_AL_INTERFACE). I have two problems with this message: a) It seems</FONT>
<BR><FONT SIZE=2>>> that this can only be used from kernel mode. b) If I understand correctly</FONT>
<BR><FONT SIZE=2>>> one has to be a bus interface for this to work. (I'm not sure that the</FONT>
<BR><FONT SIZE=2>>IPOIB</FONT>
<BR><FONT SIZE=2>>> is allowed to use such a device).</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>>Yes, query interface IRPs can only be used by kernel clients. I think it</FONT>
<BR><FONT SIZE=2>>will</FONT>
<BR><FONT SIZE=2>>be impossible to handle these IRPs in IPoIB because all PnP IRPs are</FONT>
<BR><FONT SIZE=2>>handled by</FONT>
<BR><FONT SIZE=2>>NDIS. That said the bus driver could be modified to register an interface</FONT>
<BR><FONT SIZE=2>>when</FONT>
<BR><FONT SIZE=2>>a device is started, and deregister that interface when the device is</FONT>
<BR><FONT SIZE=2>>stopped.</FONT>
<BR><FONT SIZE=2>>This still leaves some timing holes since the PDO will complete start</FONT>
<BR><FONT SIZE=2>>processing</FONT>
<BR><FONT SIZE=2>>before the upper driver does (start IRPs are processed from the bottom up)</FONT>
<BR><FONT SIZE=2>>allowing an application to issue a request before IPoIB is really ready.</FONT>
<BR><FONT SIZE=2>>The</FONT>
<BR><FONT SIZE=2>>reverse is also true for stop processing, where an application will be able</FONT>
<BR><FONT SIZE=2>>to</FONT>
<BR><FONT SIZE=2>>make requests after IPoIB stopped, but before the PDO handles the stop IRP.</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>>> 3) Use Named device objects for communicating with user and kernel</FONT>
<BR><FONT SIZE=2>>> devices. Main disadvantage of this solution seems to be that if a user</FONT>
<BR><FONT SIZE=2>>has</FONT>
<BR><FONT SIZE=2>>> opened the device, the device can not be disabled. The way I understand</FONT>
<BR><FONT SIZE=2>>it</FONT>
<BR><FONT SIZE=2>>> there will always be applications that are using Winsock and therefore</FONT>
<BR><FONT SIZE=2>>the</FONT>
<BR><FONT SIZE=2>>> driver will never go down.</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>>NDIS provides a function NdisMInitializeDevice and NdisMDeregisterDevice to</FONT>
<BR><FONT SIZE=2>>allow miniport drivers to create device objects and symbolic links for</FONT>
<BR><FONT SIZE=2>>access by</FONT>
<BR><FONT SIZE=2>>user mode. This is probably the approach we want to take - the device</FONT>
<BR><FONT SIZE=2>>object is</FONT>
<BR><FONT SIZE=2>>created from the driver's DriverEntry function, and is thus not bound to</FONT>
<BR><FONT SIZE=2>>any</FONT>
<BR><FONT SIZE=2>>particular instance of a device. The driver would have to register an</FONT>
<BR><FONT SIZE=2>>unload</FONT>
<BR><FONT SIZE=2>>handler with NdisMRegisterUnloadHandler, and deregister the device there.</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>>Applications will need to register for notifications of device removal</FONT>
<BR><FONT SIZE=2>>otherwise</FONT>
<BR><FONT SIZE=2>>the driver can never be unloaded. However, individual device instances</FONT>
<BR><FONT SIZE=2>>will</FONT>
<BR><FONT SIZE=2>>still respond properly do enable/disable requests. An application</FONT>
<BR><FONT SIZE=2>>supposedly</FONT>
<BR><FONT SIZE=2>>uses the RegisterDeviceNotification API to find out when the actual devices</FONT>
<BR><FONT SIZE=2>>go</FONT>
<BR><FONT SIZE=2>>away, but I don't quite know how it goes about using the notifications to</FONT>
<BR><FONT SIZE=2>>control when to close the device.</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>>This device would also be used by kernel clients, and the PnP notification</FONT>
<BR><FONT SIZE=2>>mechanisms there would allow proper behavior with respect to remove</FONT>
<BR><FONT SIZE=2>>requests.</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>>> 4) Use some "hack" mechanism: This might be using some UDP (or raw)</FONT>
<BR><FONT SIZE=2>>> message to send data, using "named" shared memory, and probably many</FONT>
<BR><FONT SIZE=2>>other</FONT>
<BR><FONT SIZE=2>>> ideas. I fill that this should only be used if we fill that nothing else</FONT>
<BR><FONT SIZE=2>>> works.</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>>I think if we go the hack route, we should provide a notification between</FONT>
<BR><FONT SIZE=2>>IPoIB</FONT>
<BR><FONT SIZE=2>>and KAL, and add an entry point in KAL for clients to use. I'm leaning</FONT>
<BR><FONT SIZE=2>>against</FONT>
<BR><FONT SIZE=2>>doing this, just because it seems too ugly.</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>>> If I remember well from the past, we thought about implementing the</FONT>
<BR><FONT SIZE=2>>> interface as the second type (interfaces that drivers can export in</FONT>
<BR><FONT SIZE=2>>response</FONT>
<BR><FONT SIZE=2>>> to an IRP_MN_QUERY_INTERFACE). This has the drawback that it can not be</FONT>
<BR><FONT SIZE=2>>used</FONT>
<BR><FONT SIZE=2>>> from user mode. This is fine with the SDP but will be a problem for DAPL</FONT>
<BR><FONT SIZE=2>>and</FONT>
<BR><FONT SIZE=2>>> winsock direct.</FONT>
<BR><FONT SIZE=2>>></FONT>
<BR><FONT SIZE=2>>> Should we go on number 2? Or should we think about something else?</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>>I think number 3 is the way to go due to the issues with handling PnP IRPs</FONT>
<BR><FONT SIZE=2>>within an NDIS miniport. Initially, only SDP would open the device,</FONT>
<BR><FONT SIZE=2>>register</FONT>
<BR><FONT SIZE=2>>for PnP notifications, and issue IOCTL requests to that device object.</FONT>
<BR><FONT SIZE=2>>Longer</FONT>
<BR><FONT SIZE=2>>term, the access layer might do that internally and provide higher level CM</FONT>
<BR><FONT SIZE=2>>functionality similar to what's being discussed for the Linux stack for use</FONT>
<BR><FONT SIZE=2>>by</FONT>
<BR><FONT SIZE=2>>all IP-based ULPs (SDP, WSD, DAPL). Of course, we'd have to define that</FONT>
<BR><FONT SIZE=2>>higher</FONT>
<BR><FONT SIZE=2>>level API, but that's an exercise for some other time.</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>>Using IOCTLs seems like the right way to go, as we'll be able to leverage</FONT>
<BR><FONT SIZE=2>>the</FONT>
<BR><FONT SIZE=2>>same IOCTL processing mechanisms when we start using this from user-mode.</FONT>
<BR><FONT SIZE=2>>The</FONT>
<BR><FONT SIZE=2>>implementation should ensure that the IOCTLs can be invoked at</FONT>
<BR><FONT SIZE=2>>DISPATCH_LEVEL to</FONT>
<BR><FONT SIZE=2>>eliminate constraints on kernel clients.</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>>What functionality should IPoIB provide? Should it provide:</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>>1. IP to path</FONT>
<BR><FONT SIZE=2>>2. Ethernet MAC to path</FONT>
<BR><FONT SIZE=2>>3. IP to GID</FONT>
<BR><FONT SIZE=2>>4. Ethernet MAC to GID</FONT>
<BR><FONT SIZE=2>>5. Get all locally assigned IP addresses</FONT>
<BR><FONT SIZE=2>>6. Path validation (e.g. given an IP pair and a path, return validity of</FONT>
<BR><FONT SIZE=2>>path)</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>>WSD would care most about 1, 5, and 6. DAPL probably only cares about 1</FONT>
<BR><FONT SIZE=2>>and 6.</FONT>
<BR><FONT SIZE=2>>What about SDP?</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>>Does that make sense?</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>>- Fab</FONT>
</P>
</BODY>
</HTML>