<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=US-ASCII">
<META NAME="Generator" CONTENT="MS Exchange Server version 5.5.2654.45">
<TITLE>RE: Geting remote and locale ip addresses - the functionality</TITLE>
</HEAD>
<BODY>
<P><FONT SIZE=2>I believe that we are closing on the interface, please let me know if there are still things that you don't agree about. </FONT></P>
<P><FONT SIZE=2>Thanks</FONT>
<BR><FONT SIZE=2>Tzachi</FONT>
</P>
<P><FONT SIZE=2>>-----Original Message-----</FONT>
<BR><FONT SIZE=2>>From: Fab Tillier [<A HREF="mailto:ftillier@silverstorm.com">mailto:ftillier@silverstorm.com</A>]</FONT>
<BR><FONT SIZE=2>>Sent: Thursday, September 08, 2005 1:33 AM</FONT>
<BR><FONT SIZE=2>>To: 'Tzachi Dar'; openib-windows@openib.org</FONT>
<BR><FONT SIZE=2>>Subject: RE: Geting remote and locale ip addresses - the functionality</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>>> From: Tzachi Dar [<A HREF="mailto:tzachid@mellanox.co.il">mailto:tzachid@mellanox.co.il</A>]</FONT>
<BR><FONT SIZE=2>>> Sent: Wednesday, September 07, 2005 1:24 PM</FONT>
<BR><FONT SIZE=2>>></FONT>
<BR><FONT SIZE=2>>> >From: Fab Tillier [<A HREF="mailto:ftillier@silverstorm.com">mailto:ftillier@silverstorm.com</A>]</FONT>
<BR><FONT SIZE=2>>> >Sent: Wednesday, September 07, 2005 9:01 PM</FONT>
<BR><FONT SIZE=2>>> ></FONT>
<BR><FONT SIZE=2>>> >> From: Tzachi Dar [<A HREF="mailto:tzachid@mellanox.co.il">mailto:tzachid@mellanox.co.il</A>]</FONT>
<BR><FONT SIZE=2>>> >> Sent: Wednesday, September 07, 2005 2:46 AM</FONT>
<BR><FONT SIZE=2>>> >></FONT>
<BR><FONT SIZE=2>>> >> The second problem is getting the remote GID from the remote IP. This</FONT>
<BR><FONT SIZE=2>>> >> information is only known to the IPOIB module. After doing an arp (can</FONT>
<BR><FONT SIZE=2>>be</FONT>
<BR><FONT SIZE=2>>> >> done easily from user mode and we should have the IP translated to the</FONT>
<BR><FONT SIZE=2>>> >> remote MAC). To my understanding, all information should already be in</FONT>
<BR><FONT SIZE=2>>> >> the IPOIB driver as an "endpt" (the function __endpt_mgr_ref should</FONT>
<BR><FONT SIZE=2>>> >> return it immediately).</FONT>
<BR><FONT SIZE=2>>> ></FONT>
<BR><FONT SIZE=2>>> >An alternative to adding IP support to IPoIB is to use the network stack</FONT>
<BR><FONT SIZE=2>>to</FONT>
<BR><FONT SIZE=2>>> >get us from IP to Ethernet MAC, and then perform a lookup based on that.</FONT>
<BR><FONT SIZE=2>>> >It is a bit more cumbersome, but certainly possible.</FONT>
<BR><FONT SIZE=2>>></FONT>
<BR><FONT SIZE=2>>> I believe that this is the straight forward way and this is what should</FONT>
<BR><FONT SIZE=2>>be</FONT>
<BR><FONT SIZE=2>>> used. Since there is a function that converts the remote IP to a mac</FONT>
<BR><FONT SIZE=2>>addresses,</FONT>
<BR><FONT SIZE=2>>> I believe that we should use it and do the remote query based on this arp</FONT>
<BR><FONT SIZE=2>>> addresses.</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>>I think this makes sense. We'll just have to do some work with the headers</FONT>
<BR><FONT SIZE=2>>in</FONT>
<BR><FONT SIZE=2>>the DDK to allow user-mode clients to use the IP Helper functions. This is</FONT>
<BR><FONT SIZE=2>>probably minimal. I think it's just a single header that's missing. Now,</FONT>
<BR><FONT SIZE=2>>if we</FONT>
<BR><FONT SIZE=2>>decide to depend on the IP helper library, should we just use that to</FONT>
<BR><FONT SIZE=2>>enumerate</FONT>
<BR><FONT SIZE=2>>IP addresses rather than providing our own mechanism?</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>It seems that we will have to ask people to install the platform SDK (royalties free) since these headers should be used also by the SDP user mode component and MPI. An alternative to this is to reverse engineer this component and create some header ourselves.</FONT></P>
<P><FONT SIZE=2>Unfortunately the IP helper API require a lot of work to get everything done. </FONT>
</P>
<BR>
<BR>
<P><FONT SIZE=2>>> >> This also rises the question of doing it sync or non sync</FONT>
<BR><FONT SIZE=2>>> >> (which will make things even more complicated).</FONT>
<BR><FONT SIZE=2>>> ></FONT>
<BR><FONT SIZE=2>>> >Assuming we're dealing with IRPs here and not a direct call interface,</FONT>
<BR><FONT SIZE=2>>sync</FONT>
<BR><FONT SIZE=2>>> >or async is trivial - all we need to do is follow IRP processing rules.</FONT>
<BR><FONT SIZE=2>>If</FONT>
<BR><FONT SIZE=2>>> >the request is going to take some time we need to mark the IRP pending</FONT>
<BR><FONT SIZE=2>>and</FONT>
<BR><FONT SIZE=2>>> >return (this is critical to allow clients to call this at</FONT>
<BR><FONT SIZE=2>>DISPATCH_LEVEL).</FONT>
<BR><FONT SIZE=2>>> >Once the request completes, complete the IRP. It is then up to the</FONT>
<BR><FONT SIZE=2>>caller</FONT>
<BR><FONT SIZE=2>>> >to decide whether to block waiting for a completion or use I/O</FONT>
<BR><FONT SIZE=2>>completion</FONT>
<BR><FONT SIZE=2>>> >notifications.</FONT>
<BR><FONT SIZE=2>>> ></FONT>
<BR><FONT SIZE=2>>> >I would expect that for lookups into the IPoIB adapter's internal cache,</FONT>
<BR><FONT SIZE=2>>> >the operations would pretty much always complete immediately. If we</FONT>
<BR><FONT SIZE=2>>ever</FONT>
<BR><FONT SIZE=2>>> >add logic in IPoIB to send ARPs to resolve missing entries, asynchronous</FONT>
<BR><FONT SIZE=2>>> >processing will likely be necessary. In any case, the only requirement</FONT>
<BR><FONT SIZE=2>>> >on us is that we don't perform any blocking operations in any of the</FONT>
<BR><FONT SIZE=2>>IOCTL</FONT>
<BR><FONT SIZE=2>>> >paths.</FONT>
<BR><FONT SIZE=2>>></FONT>
<BR><FONT SIZE=2>>> The function that will convert the remote IP to the remote mac will very</FONT>
<BR><FONT SIZE=2>>> likely do the arp by itself, so I don't expect any blocking operation.</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>>I think that's right. Regardless, any IRP handling we add must support</FONT>
<BR><FONT SIZE=2>>kernel</FONT>
<BR><FONT SIZE=2>>clients making requests at DISPATCH. That's really my only firm</FONT>
<BR><FONT SIZE=2>>requirement.</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>You will be able to do the lookup at DISPATCH level.</FONT>
</P>
<P><FONT SIZE=2>>> >> So to summarize:</FONT>
<BR><FONT SIZE=2>>> >> 1) Get all locale ip addresses, We can do it or not do it, I</FONT>
<BR><FONT SIZE=2>>> >> recommend to do it in the IPOIB. As the information changes, it would</FONT>
<BR><FONT SIZE=2>>> >> be better to create some notification mechanism, I suggest not</FONT>
<BR><FONT SIZE=2>>> >> implementing this mechanism at start.</FONT>
<BR><FONT SIZE=2>>> ></FONT>
<BR><FONT SIZE=2>>> >IPoIB already has this information, so I agree we should provide a way</FONT>
<BR><FONT SIZE=2>>to</FONT>
<BR><FONT SIZE=2>>> >get to it - the WSD provider certainly needs it. We don't need to</FONT>
<BR><FONT SIZE=2>>> >implement any notification mechanisms for when the addresses change,</FONT>
<BR><FONT SIZE=2>>> >as the existing stack already handles that. Specifically, it doesn't</FONT>
<BR><FONT SIZE=2>>> >matter for DAPL case because DAPL is static. For WSD, the switch</FONT>
<BR><FONT SIZE=2>>already</FONT>
<BR><FONT SIZE=2>>> >polls the providers when it detects that an update is needed.</FONT>
<BR><FONT SIZE=2>>> ></FONT>
<BR><FONT SIZE=2>>> >I think we probably want two calls here. The first would return all CA</FONT>
<BR><FONT SIZE=2>>and</FONT>
<BR><FONT SIZE=2>>> >port GUIDs in use by IPoIB, with no input parameter. The output would</FONT>
<BR><FONT SIZE=2>>be</FONT>
<BR><FONT SIZE=2>>> >something like an array <CA GUID, PORT NUMBER, PORT GUID> tupples. This</FONT>
<BR><FONT SIZE=2>>> >allows a client to open the CA (using the CA GUID), create a QP bound to</FONT>
<BR><FONT SIZE=2>>> >the proper port (using the port number), and query for IP addresses</FONT>
<BR><FONT SIZE=2>>(using</FONT>
<BR><FONT SIZE=2>>> >the port GUID).</FONT>
<BR><FONT SIZE=2>>> ></FONT>
<BR><FONT SIZE=2>>> >The second call would take as input a port GUID, and return an array of</FONT>
<BR><FONT SIZE=2>>IP</FONT>
<BR><FONT SIZE=2>>> >addresses assigned to that port. The array entries should probably be</FONT>
<BR><FONT SIZE=2>>16</FONT>
<BR><FONT SIZE=2>>> >bytes to accommodate IPv6 addresses, and use the standard method of</FONT>
<BR><FONT SIZE=2>>storing</FONT>
<BR><FONT SIZE=2>>> >IPv4 addresses (prefix with zero).</FONT>
<BR><FONT SIZE=2>>> ></FONT>
<BR><FONT SIZE=2>>> >Does that make sense?</FONT>
<BR><FONT SIZE=2>>></FONT>
<BR><FONT SIZE=2>>> This does make sense, however there is one thing that we might consider</FONT>
<BR><FONT SIZE=2>>> changing: it seems to me that the application will first do the first</FONT>
<BR><FONT SIZE=2>>part and</FONT>
<BR><FONT SIZE=2>>> later do a query on all ports to get their IP's. Therefore I believe that</FONT>
<BR><FONT SIZE=2>>we</FONT>
<BR><FONT SIZE=2>>> can have one function that returns all this information and save some</FONT>
<BR><FONT SIZE=2>>time</FONT>
<BR><FONT SIZE=2>>> passing from user mode to kernel. I'm not really sure if that code would</FONT>
<BR><FONT SIZE=2>>look</FONT>
<BR><FONT SIZE=2>>> better.</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>>I initially thought a single call would work well. It does complicate the</FONT>
<BR><FONT SIZE=2>>IOCTL</FONT>
<BR><FONT SIZE=2>>output buffer structure, but is certainly something that can be done. The</FONT>
<BR><FONT SIZE=2>>complications come from the fact that the "records" in the output buffer</FONT>
<BR><FONT SIZE=2>>are no</FONT>
<BR><FONT SIZE=2>>longer fixed size, so the size of each record (and/or offset to the next</FONT>
<BR><FONT SIZE=2>>record)</FONT>
<BR><FONT SIZE=2>>must be explicitly stated as part of the record:</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>>struct IPOIB_AT_PORT_RECORD</FONT>
<BR><FONT SIZE=2>>{</FONT>
<BR><FONT SIZE=2>> ULONG Version;</FONT>
<BR><FONT SIZE=2>> ULONG Size; //!< Total size, including IP</FONT>
<BR><FONT SIZE=2>>addresses.</FONT>
<BR><FONT SIZE=2>> ULONG OffsetNextPort; //!< From start of</FONT>
<BR><FONT SIZE=2>>structure.</FONT>
<BR><FONT SIZE=2>> ULONG NumIps;</FONT>
<BR><FONT SIZE=2>> UINT64 PortGuid;</FONT>
<BR><FONT SIZE=2>> int_addr IpAddr[1];</FONT>
<BR><FONT SIZE=2>>};</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>>struct IPOIB_AT_CA_RECORD</FONT>
<BR><FONT SIZE=2>>{</FONT>
<BR><FONT SIZE=2>> ULONG Version;</FONT>
<BR><FONT SIZE=2>> ULONG Size; //!< Total size, including</FONT>
<BR><FONT SIZE=2>>subrecords.</FONT>
<BR><FONT SIZE=2>> ULONG OffsetNextCa; //!< From start of</FONT>
<BR><FONT SIZE=2>>structure.</FONT>
<BR><FONT SIZE=2>> ULONG NumPorts;</FONT>
<BR><FONT SIZE=2>> UINT64 CaGuid;</FONT>
<BR><FONT SIZE=2>> IPOIB_AT_PORT_RECORD FirstPort;</FONT>
<BR><FONT SIZE=2>>};</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>>struct IPOIB_AT_GET_LOCAL_IP_OUT</FONT>
<BR><FONT SIZE=2>>{</FONT>
<BR><FONT SIZE=2>> ULONG Version;</FONT>
<BR><FONT SIZE=2>> ULONG Size; //!< Total size, including</FONT>
<BR><FONT SIZE=2>>subrecords.</FONT>
<BR><FONT SIZE=2>> IPOIB_AT_CA_RECORD FirstCa;</FONT>
<BR><FONT SIZE=2>>};</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>>I don't know if you would be able to measure the difference between</FONT>
<BR><FONT SIZE=2>>performing</FONT>
<BR><FONT SIZE=2>>multiple simple IOCTLs versus performing a single complicated IOCTL.</FONT>
<BR><FONT SIZE=2>>Further,</FONT>
<BR><FONT SIZE=2>>the multiple IOCTL approach allows a user to only request IP information</FONT>
<BR><FONT SIZE=2>>for a</FONT>
<BR><FONT SIZE=2>>particular port - in case they don't care about any of the other ports.</FONT>
<BR><FONT SIZE=2>>Note</FONT>
<BR><FONT SIZE=2>>that the version fields can probably be eliminated but won't result in</FONT>
<BR><FONT SIZE=2>>smaller</FONT>
<BR><FONT SIZE=2>>records due to padding. The size fields could also be eliminated, but</FONT>
<BR><FONT SIZE=2>>would</FONT>
<BR><FONT SIZE=2>>result in extra processing when filling the buffer.</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>>Personally I think the multi-IOCTL method will be simpler to implement</FONT>
<BR><FONT SIZE=2>>while</FONT>
<BR><FONT SIZE=2>>providing the client with extra flexibility.</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>We will use the multi-IOCTL method to get the data.</FONT>
</P>
<P><FONT SIZE=2>>> >> 2) Get remote IP - we must implement.</FONT>
<BR><FONT SIZE=2>>> ></FONT>
<BR><FONT SIZE=2>>> >Assuming this takes a source and destination IP as input and returns a</FONT>
<BR><FONT SIZE=2>>> >source and destination GID as output, I agree.</FONT>
<BR><FONT SIZE=2>>> ></FONT>
<BR><FONT SIZE=2>>> >How does something like this for the IOCTL input and output buffers</FONT>
<BR><FONT SIZE=2>>sound:</FONT>
<BR><FONT SIZE=2>>> ></FONT>
<BR><FONT SIZE=2>>> >struct _IPOIB_AT_IN</FONT>
<BR><FONT SIZE=2>>> >{</FONT>
<BR><FONT SIZE=2>>> > UCHAR SrcIp[16];</FONT>
<BR><FONT SIZE=2>>> > UCHAR DstIp[16];</FONT>
<BR><FONT SIZE=2>>> >};</FONT>
<BR><FONT SIZE=2>>> >struct _IPOIB_AT_OUT</FONT>
<BR><FONT SIZE=2>>> >{</FONT>
<BR><FONT SIZE=2>>> > GID SrcGid;</FONT>
<BR><FONT SIZE=2>>> > GID DstGid;</FONT>
<BR><FONT SIZE=2>>> >};</FONT>
<BR><FONT SIZE=2>>></FONT>
<BR><FONT SIZE=2>>> As I said before I believe that the functionality that we are really</FONT>
<BR><FONT SIZE=2>>looking</FONT>
<BR><FONT SIZE=2>>> for is the remote mac addresses, translated into a remote lid. (This is</FONT>
<BR><FONT SIZE=2>>the</FONT>
<BR><FONT SIZE=2>>> information that we have). So the interface should look like:</FONT>
<BR><FONT SIZE=2>>> struct _IPOIB_AT_IN</FONT>
<BR><FONT SIZE=2>>> {</FONT>
<BR><FONT SIZE=2>>> UCHAR DstMac[6]; // Do we want this longer ?</FONT>
<BR><FONT SIZE=2>>> };</FONT>
<BR><FONT SIZE=2>>> struct _IPOIB_AT_OUT</FONT>
<BR><FONT SIZE=2>>> {</FONT>
<BR><FONT SIZE=2>>> GID DstGid;</FONT>
<BR><FONT SIZE=2>>> };</FONT>
<BR><FONT SIZE=2>>> Does this make sense?</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>>Do we need to specify as part of the input buffer which instance to perform</FONT>
<BR><FONT SIZE=2>>the</FONT>
<BR><FONT SIZE=2>>lookup for? Or will the destination MAC address always be unique,</FONT>
<BR><FONT SIZE=2>>regardless of</FONT>
<BR><FONT SIZE=2>>how the fabric is configured? This depends on the algorithm used to</FONT>
<BR><FONT SIZE=2>>generate</FONT>
<BR><FONT SIZE=2>>the LAA MAC addresses reported to the OS. I think right now they are</FONT>
<BR><FONT SIZE=2>>unique.</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>I assume that the MACs are unique, If this is not the case other things won't work eitheir.</FONT>
</P>
<P><FONT SIZE=2>>Also, how does a client get the source GID?</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>The GID is needed for the IB_QUERY_PATH_REC_BY_GIDS. I believe that using IB_DEFAULT_SUBNET_PREFIX should be enough for this query.</FONT></P>
<P><FONT SIZE=2>>It might make sense to pass in the local port GUID (or CA GUID and port</FONT>
<BR><FONT SIZE=2>>number)</FONT>
<BR><FONT SIZE=2>>to better qualify the request and provide the source GID as output.</FONT>
<BR><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>>Thoughts?</FONT>
<BR><FONT SIZE=2>This seems to me like an extra thing to check, while it will get checked by QUERY_PATH_REC in any case, so I don't think that there is a reason to bother.</FONT></P>
<P><FONT SIZE=2>></FONT>
<BR><FONT SIZE=2>>- Fab</FONT>
</P>
</BODY>
</HTML>