[libfabric-users] Linux verbs / Windows netdir Interoperability

Hefty, Sean sean.hefty at intel.com
Fri Nov 5 12:00:22 PDT 2021

> So, after getting our use case working from Windows to Windows, I dove into attempting
> Linux/Windows interoperability with Linux being the sender and Windows being the
> receiver. The two endpoints connected after accounting for the fact that the verbs
> library silently adds and removes a byte at the start of the CM data, while the netdir
> provider does not.

Interoperability wasn't considered here.  Are you using connected or RDM endpoints?  The primary use case is RDM.

> However, the fi_sendv from Linux failed on the netdir side with an ND_DATA_OVERRUN
> error status. After digging into the netdir provider code a bit, I think I see the
> source of the problem. It appears that the receive buffer provided in the call to
> fi_recvv is not immediately sent to the ND2 implementation, but is queued inside the
> netdir provider. It appears that the netdir provider expects a small incoming message
> indicating that a large message transfer is being requested. When processing this small
> initiation message, the receive buffer is then processed and passed to the ND2
> implementation. Then the large message is transferred and processed.
> So, it appears that some effort will be needed to make changes to the netdir provider
> in order to support our desired use case. My manager is on board with our making the
> effort to achieve our desired use case and contributing the code back upstream. Our
> concern is that since none of us are particularly experienced with libfabric or Network
> Direct, we don’t know whether there might be some hard limitation in the Network Direct
> API that would stop us dead in our tracks from achieving interoperability between Linux
> verbs and Windows netdir.

That would be great.  I'm pretty sure ND to verbs interoperability has been achieved in the past, just not through libfabric.  I don't anticipate changes to the netdir provider wire protocol to be an issue, or that it'll need to be backwards compatible.

> We’re wondering whether anyone here knows why the netdir provider might have been
> designed the way that it is. Or even better, if the original implementers might be
> available for consultation about their design choices and knowledge about what
> limitations we would face in attempting to make changes to achieve our use case.

I think the original authors actually work for Nvidia now.  The target (only?) application for the provider is Intel MPI.  So, the MPI implementation and use case drove the implementation.

- Sean

More information about the Libfabric-users mailing list