[libfabric-users] Linux verbs / Windows netdir Interoperability
sean.hefty at intel.com
Fri Nov 5 12:00:22 PDT 2021
> So, after getting our use case working from Windows to Windows, I dove into attempting
> Linux/Windows interoperability with Linux being the sender and Windows being the
> receiver. The two endpoints connected after accounting for the fact that the verbs
> library silently adds and removes a byte at the start of the CM data, while the netdir
> provider does not.
Interoperability wasn't considered here. Are you using connected or RDM endpoints? The primary use case is RDM.
> However, the fi_sendv from Linux failed on the netdir side with an ND_DATA_OVERRUN
> error status. After digging into the netdir provider code a bit, I think I see the
> source of the problem. It appears that the receive buffer provided in the call to
> fi_recvv is not immediately sent to the ND2 implementation, but is queued inside the
> netdir provider. It appears that the netdir provider expects a small incoming message
> indicating that a large message transfer is being requested. When processing this small
> initiation message, the receive buffer is then processed and passed to the ND2
> implementation. Then the large message is transferred and processed.
> So, it appears that some effort will be needed to make changes to the netdir provider
> in order to support our desired use case. My manager is on board with our making the
> effort to achieve our desired use case and contributing the code back upstream. Our
> concern is that since none of us are particularly experienced with libfabric or Network
> Direct, we don’t know whether there might be some hard limitation in the Network Direct
> API that would stop us dead in our tracks from achieving interoperability between Linux
> verbs and Windows netdir.
That would be great. I'm pretty sure ND to verbs interoperability has been achieved in the past, just not through libfabric. I don't anticipate changes to the netdir provider wire protocol to be an issue, or that it'll need to be backwards compatible.
> We’re wondering whether anyone here knows why the netdir provider might have been
> designed the way that it is. Or even better, if the original implementers might be
> available for consultation about their design choices and knowledge about what
> limitations we would face in attempting to make changes to achieve our use case.
I think the original authors actually work for Nvidia now. The target (only?) application for the provider is Intel MPI. So, the MPI implementation and use case drove the implementation.
More information about the Libfabric-users