[ofw] Use of I/O completion port with Winverbs

Mon Aug 19 05:31:20 PDT 2013

This is a topic of great interest to us.

Is it correct to say that, today, ND is the only available API on Win(8/2012) that will interop with Linux rdma_cm?

Similarly, with OFW for Win(8/2012) delayed (sadly) and Mellanox apparently not including rdma_cm in their Windows kit, is there no way today to utilize those "advanced" features like atomics, immediates, unreliable, etc on Win(8/2012) at all (much less interoperably)?

TIA!

- Dardo

On 08/18/2013 03:18 PM, Fab Tillier wrote:
> Ah, I didn’t realize Mellanox included the headers in their redist, ok.
>
> The docs for NDv2 are in the pipeline, but there’s quite a backlog for our tech writers so I don’t have an ETA unfortunately.  The combination of NDv1 and NDK docs should get you on the right path in the meantime, and you can always send me questions.
>
> Provider discovery is largely the same as NDv1, though you call DllGetClassObject for the IND2Provider interface directly, rather than going through the IClassFactory interface.
>
> You should be able to interop between ND and librdma code, as well as between Windows and Linux no problem.  If it doesn’t work then there’s a bug in the drivers somewhere.  Of course you’ll be limited by the capabilities of NDv2 (no remote atomic, only reliable connected, no immediate data, etc.)
>
> -Fab
>
> *From:* Christoph Müller [mailto:christoph.mueller at visus.uni-stuttgart.de]
> *Sent:* Sunday, August 18, 2013 12:01 PM
> *To:* Fab Tillier
> *Cc:* 'ofw at lists.openfabrics.org'
> *Subject:* AW: Use of I/O completion port with Winverbs
>
> Hi Fab,
>
> thanks. I already got access to the IDN2Provider on Friday - the headers are also included in Mellanox' distribution just as Tzachi said. The problem on my side was that I was looking for some import libraries. I finally found some NDv1 documentation on MSDN explaining how to get the root object via the Winsock provider catalogue. However, there does not seem to be any NDv2 documentation yet, but the differences seem to be subtle.
>
> As I have only limited time, I will probably not be able to rewrite all of my code at once. I think that my NDv2 code will be able to talk to librdma peers as long as I use only basic functionality like remote read/write or inline send/receive. Is that correct?
>
> Best regards,
> Christoph
>
> Von meinem Windows Phone gesendet
>
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------!
 ---
>
> *Von: *Fab Tillier <mailto:ftillier at microsoft.com>
> *Gesendet: *‎18.‎08.‎2013 07:38
> *An: *Christoph Müller <mailto:christoph.mueller at visus.uni-stuttgart.de>
> *Cc: *'ofw at lists.openfabrics.org' <mailto:ofw at lists.openfabrics.org>
> *Betreff: *RE: Use of I/O completion port with Winverbs
>
> Hi Christoph,
>
> You can get the ND header files (included in the NetworkDirect DDK zip file) here:
>
> http://www.microsoft.com/en-us/download/details.aspx?id=36043
>
> There is no dependency on the HPC Pack - you just need drivers that support ND (ideally NDv2, which the Mellanox 4.40 release does).
>
> Using ND should give you the greatest hardware independence too, though by design ND exposes the lowest common denominator functionality of iWARP and InfiniBand, so some functionality is not available (e.g. RDMA atomics).
>
> Don’t hesitate to ask if you have questions.
>
> -Fab
>
> *From:* ofw-bounces at lists.openfabrics.org <mailto:ofw-bounces at lists.openfabrics.org> [mailto:ofw-bounces at lists.openfabrics.org] *On Behalf Of *Christoph Müller
> *Sent:* Thursday, August 15, 2013 6:04 AM
> *To:* Tzachi Dar; ofw at lists.openfabrics.org <mailto:ofw at lists.openfabrics.org>
> *Subject:* Re: [ofw] Use of I/O completion port with Winverbs
>
> Thank you. I was not aware that the SDK including NDv2 is distributed with Mellanox’ drivers. In this case, I will give it a try.
>
> Thanks again,
>
> Christoph
>
> *Von:*Tzachi Dar [mailto:tzachid at mellanox.com]
> *Gesendet:* Donnerstag, 15. August 2013 14:43
> *An:* Christoph Müller; ofw at lists.openfabrics.org <mailto:ofw at lists.openfabrics.org>
> *Betreff:* RE: Use of I/O completion port with Winverbs
>
> You only need to download Mellanox drivers 4.40 for windows, and you will have nd support.
>
> http://www.mellanox.com/page/products_dyn?product_family=32&mtag=windows_sw_drivers
>
> Thanks
>
> Tzachi
>
> *From:*Christoph Müller [mailto:christoph.mueller at visus.uni-stuttgart.de]
> *Sent:* Thursday, August 15, 2013 3:32 PM
> *To:* Tzachi Dar; ofw at lists.openfabrics.org <mailto:ofw at lists.openfabrics.org>
> *Subject:* AW: Use of I/O completion port with Winverbs
>
> ndinstall is part of the HPC pack distribution? My main problem with ND is to find out how to get it...
>
> Best regards,
> Christoph
>
> Von meinem Windows Phone gesendet
>
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------!
 ---
>
> *Von: *Tzachi Dar <mailto:tzachid at mellanox.com>
> *Gesendet: *‎15.‎08.‎2013 14:28
> *An: *Christoph Müller <mailto:christoph.mueller at visus.uni-stuttgart.de>; ofw at lists.openfabrics.org <mailto:ofw at lists.openfabrics.org>
> *Betreff: *RE: Use of I/O completion port with Winverbs
>
> Please note that the windows APIs are changing from time to time and ND is probably the one that will stay longer since it is supported by MS.
>
> I don’t think that you need to install the hpc pack for it. You simply need to run ndinstall and then you can start running nd programs.
>
> Thanks
>
> Tzachi
>
> *From:*Christoph Müller [mailto:christoph.mueller at visus.uni-stuttgart.de]
> *Sent:* Thursday, August 15, 2013 3:21 PM
> *To:* Tzachi Dar; ofw at lists.openfabrics.org <mailto:ofw at lists.openfabrics.org>
> *Subject:* AW: Use of I/O completion port with Winverbs
>
> Actually, I am in the process of deciding which API to use (although I have existing librdma code). NetworkDirect indeed looks promising, but as I understand, I need HPC Pack 2012 for that which I cannot install on our cluster in the short term.
>
> Best regards,
> Christoph
>
>
> Von meinem Windows Phone gesendet
>
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------!
 ---
>
> *Von: *Tzachi Dar <mailto:tzachid at mellanox.com>
> *Gesendet: *‎15.‎08.‎2013 14:15
> *An: *Christoph Müller <mailto:christoph.mueller at visus.uni-stuttgart.de>; ofw at lists.openfabrics.org <mailto:ofw at lists.openfabrics.org>
> *Betreff: *RE: Use of I/O completion port with Winverbs
>
> can you write your application with Network Direct?
>
> If so, there are overlapped objects there that you should be able to connect to the completion ports.
>
> Thanks
>
> Tzachi
>
> *From:*ofw-bounces at lists.openfabrics.org <mailto:ofw-bounces at lists.openfabrics.org> [mailto:ofw-bounces at lists.openfabrics.org] *On Behalf Of *Christoph Müller
> *Sent:* Thursday, August 15, 2013 3:10 PM
> *To:* ofw at lists.openfabrics.org <mailto:ofw at lists.openfabrics.org>
> *Subject:* [ofw] Use of I/O completion port with Winverbs
>
> Hi all,
>
> I was wondering whether it is possible to use CreateIoCompletionPort etc. with IWVConnectEndpoints. I can see that the implementation inherits a handle from CWVBase, but I do not see how I could use this information for creating an IOCP and, of course, I do not know whether it would work. Furthermore, I would prefer continuing to use the RDMA library, but I see absolutely no way to create an IOCP here. So me questions are:
>
> Is it possible to use IOCPs with winverbs?
>
> Is it possible to use IOCPs with the RDMA library?
>
> Is there sample code for either of it?
>
> Thanks in advance,
>
> Christoph
>
> --
>
> Visualisierungsinstitut der Universität Stuttgart
>
> Allmandring 19, 70569 Stuttgart
>
> E-Mail: christoph.mueller at visus.uni-stuttgart.de <mailto:christoph.mueller at visus.uni-stuttgart.de>
>
> Telefon: (07 11) 685 - 88626
>
>
>
> _______________________________________________
> ofw mailing list
> ofw at lists.openfabrics.org
> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ofw
>