[libfabric-users] troubled by FI_SOURCE use

[Hefty, Sean]

> Just a follow up question - I was busy with other items and didn't really chase
> up on investigating the RXM and RXD providers.
> As utility providers, is there a matrix (like the very useful feature matrix)
> that says which hardware providers they are compatible with?
> For example, if I want to use RXD on top of verbs or RXM on top of verbs, or on
> top of sockets, tcp, udp - how do I know which ones are possible?

We should add a utility provider requirement matrix.

RXM will work over verbs RC and XRC, and tcp.
RXD will work over verbs UD, and udp.

> I suppose I should assume that
> * if the core provider supports MSG endpoint then RXM can run on it - NB the
> fi_provider man page mentions RXM, but doesn't really include RXD in the utility
> providers list but discusses it later).
> * if the core provider supports DGRAM endpoint then RXD can run on it.

There are other requirements beyond the EP type, plus optimizations which a utility provider will use if available.  E.g. I believe RXM requires RDMA read support.  It requires RDMA read and write if the user wants RMA ops.

> So If I need RDM endpoint type for GNI/Sockets/Verbs I can use
> GNI/rxd+udp/rxm+verbs for the 3 configurations?

Yes, this is one option.

> On an unrelated note. I was testing the libfabric (gni) code on our cray and
> doing random messages between nodes and got 6.2TB/s aggregated between 4096
> nodes. Still some tuning to do to reach max, but I'm very impressed. Well done
> to all of the libfabric developers and thank you for making our lives easier.

Thanks!

- Sean