[libfabric-users] questions on erbs provider for RDMA

[Xiong, Jianxin]

All three have Verbs underneath and work over IB or RoCE network. Which provider to use depends on the OFI endpoint type requested by the application.

The verbs provider supports FI_EP_MSG and FI_EP_DGRAM. FI_EP_MSG maps directly to Verbs RC. Each endpoint maps to an RC QP. Explicit connection setup is needed before communication can happen. Multiple endpoints are needed in order to talk to different peers. FI_EP_DGRAM maps directly to Verbs UD. It's connectionless, but is unreliable, and message size is limited by MTU size.

Ofi_rxm is a utility provider that runs on top of the verbs provider (using FI_EP_MSG type) and provide connectionless semantics (FI_EP_RDM). Under the cover, each ofi_rxm endpoint maps to verb endpoints and connection is established automatically on demand. You don't need to ask for the combination explicitly.  When the application asks for "FI_EP_RDM" endpoint type, "verbs;ofi_rxm" is selected automatically.

Ofi_rxd is a utility provider that runs on top the verbs provider (using FI_EP_DGRAM type) and present FI_EP_RDM support.  Its functionality is limited compared to ofi_rxm so it is usually not the first choice.

In summary, an application only needs to ask for the "verbs" provider for RDMA. If an application wants to manage the connection setup by itself, it can ask for ep_type FI_EP_MSG and get the "bare" verbs provider. The application is then responsible to setup up the connections between endpoints by calling fi_passive_ep(), fi_listen(), fi_connect(), and fi_accept(). If an application doesn't want to manage connection setup, it can ask for ep_type FI_EP_RDM and get the "verbs;ofi_rxm" provider. The application then needs to obtain the endpoint address with fi_getname(), exchange the addresses with peers using out-of-band mechanism, and insert the addresses into an address vector. Future communication will use addresses from the address vector as the identifier for the peer.

-Jianxin

From: Libfabric-users <libfabric-users-bounces at lists.openfabrics.org> On Behalf Of Niyaz Murshed
Sent: Tuesday, June 25, 2024 2:08 PM
To: libfabric-users at lists.openfabrics.org
Cc: nd <nd at arm.com>
Subject: [libfabric-users] questions on erbs provider for RDMA

Hi all ,

I am trying to understand the verbs provider specially difference between the below:



  1.  Verbs
  2.  Verbs;ofi_rxd
  3.  Verbs;ofi_rxm


They seem to work on 3 different protocols




FI_PROTO_RDMA_CM_IB_RC
The protocol runs over Infiniband reliable-connected queue pairs, using the RDMA CM protocol for connection establishment.


FI_PROTO_RXM
Reliable-datagram protocol implemented over message endpoints. RXM is a libfabric utility component that adds RDM endpoint semantics over MSG endpoint semantics.
FI_PROTO_RXD
Reliable-datagram protocol implemented over datagram endpoints. RXD is a libfabric utility component that adds RDM endpoint semantics over DGRAM endpoint semantics.




>From my test of FI_PROTO_RDMA_CM_IB_RC, I see in wireshark that, its RoCEv2 protocol when I test application of RDMA.
What protocol do RXM and RXD use? I see its TCP packets on the wire? Does it mean it uses TCP ?
Is it possible to use RoCEv2 for RXM and RXD?
What is the best provider to use for RDMA ?

Regards,
Niyaz
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/libfabric-users/attachments/20240625/30c21e22/attachment-0001.htm>