[ofw] Use of I/O completion port with Winverbs

[Christoph Müller]

Hi Stan,

thanks. I already got the adapters. In case I encounter a problem, I will post a specific questions.

Best regards,
Christoph

Von: Smith, Stan [mailto:stan.smith at intel.com]
Gesendet: Montag, 19. August 2013 18:12
An: Fab Tillier; Christoph Müller
Cc: 'ofw at lists.openfabrics.org'
Betreff: RE: Use of I/O completion port with Winverbs

Christoph,
  Located in the winOFED src tree under ulp\dapl2\dapl\NetworkDirect\NdHelper.cpp, specifically NdOpenAdapter().
Also in dapl_nd_device.c : dapls_ib_open_hca() could provide some NDv2 device handling insights.
W.r.t. to which ND device, NdQueryAddressList() will return a list of ND devices on the system; also see dapli_GetIPaddr().
For the DAPL case, I match a device name from dat.conf ( hca0 ) and use the device name trailing instance number as an index into the NdQueryAddressList() of ND devices.

Stan.

From: ofw-bounces at lists.openfabrics.org<mailto:ofw-bounces at lists.openfabrics.org> [mailto:ofw-bounces at lists.openfabrics.org] On Behalf Of Fab Tillier
Sent: Sunday, August 18, 2013 12:19 PM
To: Christoph Müller
Cc: 'ofw at lists.openfabrics.org'
Subject: Re: [ofw] Use of I/O completion port with Winverbs

Ah, I didn’t realize Mellanox included the headers in their redist, ok.

The docs for NDv2 are in the pipeline, but there’s quite a backlog for our tech writers so I don’t have an ETA unfortunately.  The combination of NDv1 and NDK docs should get you on the right path in the meantime, and you can always send me questions.

Provider discovery is largely the same as NDv1, though you call DllGetClassObject for the IND2Provider interface directly, rather than going through the IClassFactory interface.

You should be able to interop between ND and librdma code, as well as between Windows and Linux no problem.  If it doesn’t work then there’s a bug in the drivers somewhere.  Of course you’ll be limited by the capabilities of NDv2 (no remote atomic, only reliable connected, no immediate data, etc.)

-Fab

From: Christoph Müller [mailto:christoph.mueller at visus.uni-stuttgart.de]
Sent: Sunday, August 18, 2013 12:01 PM
To: Fab Tillier
Cc: 'ofw at lists.openfabrics.org'
Subject: AW: Use of I/O completion port with Winverbs

Hi Fab,

thanks. I already got access to the IDN2Provider on Friday - the headers are also included in Mellanox' distribution just as Tzachi said. The problem on my side was that I was looking for some import libraries. I finally found some NDv1 documentation on MSDN explaining how to get the root object via the Winsock provider catalogue. However, there does not seem to be any NDv2 documentation yet, but the differences seem to be subtle.

As I have only limited time, I will probably not be able to rewrite all of my code at once. I think that my NDv2 code will be able to talk to librdma peers as long as I use only basic functionality like remote read/write or inline send/receive. Is that correct?

Best regards,
Christoph

Von meinem Windows Phone gesendet
________________________________
Von: Fab Tillier<mailto:ftillier at microsoft.com>
Gesendet: ‎18.‎08.‎2013 07:38
An: Christoph Müller<mailto:christoph.mueller at visus.uni-stuttgart.de>
Cc: 'ofw at lists.openfabrics.org'<mailto:ofw at lists.openfabrics.org>
Betreff: RE: Use of I/O completion port with Winverbs
Hi Christoph,

You can get the ND header files (included in the NetworkDirect DDK zip file) here:
http://www.microsoft.com/en-us/download/details.aspx?id=36043

There is no dependency on the HPC Pack - you just need drivers that support ND (ideally NDv2, which the Mellanox 4.40 release does).

Using ND should give you the greatest hardware independence too, though by design ND exposes the lowest common denominator functionality of iWARP and InfiniBand, so some functionality is not available (e.g. RDMA atomics).

Don’t hesitate to ask if you have questions.
-Fab

From: ofw-bounces at lists.openfabrics.org<mailto:ofw-bounces at lists.openfabrics.org> [mailto:ofw-bounces at lists.openfabrics.org] On Behalf Of Christoph Müller
Sent: Thursday, August 15, 2013 6:04 AM
To: Tzachi Dar; ofw at lists.openfabrics.org<mailto:ofw at lists.openfabrics.org>
Subject: Re: [ofw] Use of I/O completion port with Winverbs

Thank you. I was not aware that the SDK including NDv2 is distributed with Mellanox’ drivers. In this case, I will give it a try.

Thanks again,
Christoph

Von: Tzachi Dar [mailto:tzachid at mellanox.com]
Gesendet: Donnerstag, 15. August 2013 14:43
An: Christoph Müller; ofw at lists.openfabrics.org<mailto:ofw at lists.openfabrics.org>
Betreff: RE: Use of I/O completion port with Winverbs

You only need to download Mellanox drivers 4.40 for windows, and you will have nd support.

http://www.mellanox.com/page/products_dyn?product_family=32&mtag=windows_sw_drivers

Thanks
Tzachi


From: Christoph Müller [mailto:christoph.mueller at visus.uni-stuttgart.de]
Sent: Thursday, August 15, 2013 3:32 PM
To: Tzachi Dar; ofw at lists.openfabrics.org<mailto:ofw at lists.openfabrics.org>
Subject: AW: Use of I/O completion port with Winverbs

ndinstall is part of the HPC pack distribution? My main problem with ND is to find out how to get it...

Best regards,
Christoph

Von meinem Windows Phone gesendet
________________________________
Von: Tzachi Dar<mailto:tzachid at mellanox.com>
Gesendet: ‎15.‎08.‎2013 14:28
An: Christoph Müller<mailto:christoph.mueller at visus.uni-stuttgart.de>; ofw at lists.openfabrics.org<mailto:ofw at lists.openfabrics.org>
Betreff: RE: Use of I/O completion port with Winverbs
Please note that the windows APIs are changing from time to time and ND is probably the one that will stay longer since it is supported by MS.

I don’t think that you need to install the hpc pack for it. You simply need to run ndinstall and then you can start running nd programs.

Thanks
Tzachi

From: Christoph Müller [mailto:christoph.mueller at visus.uni-stuttgart.de]
Sent: Thursday, August 15, 2013 3:21 PM
To: Tzachi Dar; ofw at lists.openfabrics.org<mailto:ofw at lists.openfabrics.org>
Subject: AW: Use of I/O completion port with Winverbs

Actually, I am in the process of deciding which API to use (although I have existing librdma code). NetworkDirect indeed looks promising, but as I understand, I need HPC Pack 2012 for that which I cannot install on our cluster in the short term.

Best regards,
Christoph


Von meinem Windows Phone gesendet
________________________________
Von: Tzachi Dar<mailto:tzachid at mellanox.com>
Gesendet: ‎15.‎08.‎2013 14:15
An: Christoph Müller<mailto:christoph.mueller at visus.uni-stuttgart.de>; ofw at lists.openfabrics.org<mailto:ofw at lists.openfabrics.org>
Betreff: RE: Use of I/O completion port with Winverbs
can you write your application with Network Direct?
If so, there are overlapped objects there that you should be able to connect to the completion ports.

Thanks
Tzachi

From: ofw-bounces at lists.openfabrics.org<mailto:ofw-bounces at lists.openfabrics.org> [mailto:ofw-bounces at lists.openfabrics.org] On Behalf Of Christoph Müller
Sent: Thursday, August 15, 2013 3:10 PM
To: ofw at lists.openfabrics.org<mailto:ofw at lists.openfabrics.org>
Subject: [ofw] Use of I/O completion port with Winverbs

Hi all,

I was wondering whether it is possible to use CreateIoCompletionPort etc. with IWVConnectEndpoints. I can see that the implementation inherits a handle from CWVBase, but I do not see how I could use this information for creating an IOCP and, of course, I do not know whether it would work. Furthermore, I would prefer continuing to use the RDMA library, but I see absolutely no way to create an IOCP here. So me questions are:

Is it possible to use IOCPs with winverbs?
Is it possible to use IOCPs with the RDMA library?
Is there sample code for either of it?

Thanks in advance,
Christoph
--
Visualisierungsinstitut der Universität Stuttgart
Allmandring 19, 70569 Stuttgart
E-Mail: christoph.mueller at visus.uni-stuttgart.de<mailto:christoph.mueller at visus.uni-stuttgart.de>
Telefon: (07 11) 685 - 88626

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/ofw/attachments/20130819/377b69c2/attachment.html>