[Openib-windows] RE: geting remote and locale ip addresses

[Fab Tillier]

> From: Tzachi Dar [mailto:tzachid at mellanox.co.il]
> Sent: Monday, September 05, 2005 3:11 PM
> 
> Hi Fab,
> 
> I have started looking at the implementation of this functionality in the
> IPOIB driver. It seems that the first thing that should be decided is the
> way that the IPOIB driver will expose some way to talk with other components
> (also from the user mode).
> 
> To me it seems that there are 4 ways to achieve this task:
> 
> 1)     It seems to me that "Device Interface Classes" as they are called in
> the MSDN (search for this strings in MSDN in "titles only") are the way that
> MSDN recommends to do the task. (Do not confuse device interfaces with the
> interfaces that drivers can export in response to an IRP_MN_QUERY_INTERFACE
> request. That IRP is used to pass routine entry points between kernel-mode
> drivers.) On the other hand these functions don't seem to allow any way of
> custom connection and are therefore (probably) can not be used.

The IRP_MN_QUERY_INTERFACE method can be used to find a device that supports a
particular interface using the SetupDiXxx functions.  There are also
notification mechanisms to allow an application to release the device when the
interface becomes unavailable.  Once the device is resolved, an application
would send requests via IOCTLs.

This isn't a bad method, but still requires the NDIS miniport to handle IOCTLs.

> 2)     Use the IRP_MN_QUERY_INTERFACE way. It seems that this way is used by
> our stack to pass interfaces from one component to another (for example the
> GUID_IB_AL_INTERFACE). I have two problems with this message: a) It seems
> that this can only be used from kernel mode. b) If I understand correctly
> one has to be a bus interface for this to work. (I'm not sure that the IPOIB
> is allowed to use such a device).

Yes, query interface IRPs can only be used by kernel clients.  I think it will
be impossible to handle these IRPs in IPoIB because all PnP IRPs are handled by
NDIS.  That said the bus driver could be modified to register an interface when
a device is started, and deregister that interface when the device is stopped.
This still leaves some timing holes since the PDO will complete start processing
before the upper driver does (start IRPs are processed from the bottom up)
allowing an application to issue a request before IPoIB is really ready.  The
reverse is also true for stop processing, where an application will be able to
make requests after IPoIB stopped, but before the PDO handles the stop IRP.

> 3)     Use Named device objects for communicating with user and kernel
> devices. Main disadvantage of this solution seems to be that if a user has
> opened the device, the device can not be disabled. The way I understand it
> there will always be applications that are using Winsock and therefore the
> driver will never go down.

NDIS provides a function NdisMInitializeDevice and NdisMDeregisterDevice to
allow miniport drivers to create device objects and symbolic links for access by
user mode.  This is probably the approach we want to take - the device object is
created from the driver's DriverEntry function, and is thus not bound to any
particular instance of a device.  The driver would have to register an unload
handler with NdisMRegisterUnloadHandler, and deregister the device there.

Applications will need to register for notifications of device removal otherwise
the driver can never be unloaded.  However, individual device instances will
still respond properly do enable/disable requests.  An application supposedly
uses the RegisterDeviceNotification API to find out when the actual devices go
away, but I don't quite know how it goes about using the notifications to
control when to close the device.

This device would also be used by kernel clients, and the PnP notification
mechanisms there would allow proper behavior with respect to remove requests.

> 4)     Use some "hack" mechanism: This might be using some UDP (or raw)
> message to send data, using "named" shared memory, and probably many other
> ideas. I fill that this should only be used if we fill that nothing else
> works.

I think if we go the hack route, we should provide a notification between IPoIB
and KAL, and add an entry point in KAL for clients to use.  I'm leaning against
doing this, just because it seems too ugly.

> If I remember well from the past, we thought about implementing the
> interface as the second type (interfaces that drivers can export in response
> to an IRP_MN_QUERY_INTERFACE). This has the drawback that it can not be used
> from user mode. This is fine with the SDP but will be a problem for DAPL and
> winsock direct.
> 
> Should we go on number 2? Or should we think about something else?

I think number 3 is the way to go due to the issues with handling PnP IRPs
within an NDIS miniport.  Initially, only SDP would open the device, register
for PnP notifications, and issue IOCTL requests to that device object.  Longer
term, the access layer might do that internally and provide higher level CM
functionality similar to what's being discussed for the Linux stack for use by
all IP-based ULPs (SDP, WSD, DAPL).  Of course, we'd have to define that higher
level API, but that's an exercise for some other time.

Using IOCTLs seems like the right way to go, as we'll be able to leverage the
same IOCTL processing mechanisms when we start using this from user-mode.  The
implementation should ensure that the IOCTLs can be invoked at DISPATCH_LEVEL to
eliminate constraints on kernel clients.

What functionality should IPoIB provide?  Should it provide:

1. IP to path
2. Ethernet MAC to path
3. IP to GID
4. Ethernet MAC to GID
5. Get all locally assigned IP addresses
6. Path validation (e.g. given an IP pair and a path, return validity of path)

WSD would care most about 1, 5, and 6.  DAPL probably only cares about 1 and 6.
What about SDP?
 
Does that make sense?

- Fab