[libfabric-users] Using the verbs provider with manual address exchange

[Ilango, Arun]

Hi Piotr,

To clarify a few things,
* The rc_pingong.c example in fabtests uses OFI connection management APIs (https://ofiwg.github.io/libfabric/master/man/fi_cm.3.html) to establish a connection between two FI_EP_MSG endpoints.
* The OFI verbs provider internally uses rdmacm to connect IB queue pairs and is not visible to apps using OFI.
* AVs are used only for unconnected endpoint types (FI_EP_RDM and FI_EP_DGRAM).

To run a simple hello program with a client and server, the client needs to know the server's IPoIB address. You can take a look at the example in simple/msg.c in fabtests. This example uses FI_EP_MSG endpoint type. The apps need not exchange IB addresses since that would be handled by OFI verbs. For now, OFI verbs support only tagged messages (FI_TAGGED capability flag) for FI_EP_RDM endpoint type. Please take a look at benchmarks/rdm_tagged_pingpong.c for an example.

I hope this helps. Let me know if you have more questions.

Thanks,
Arun.

From: Libfabric-users [mailto:libfabric-users-bounces at lists.openfabrics.org] On Behalf Of LESNICKI, PIOTR
Sent: Thursday, September 01, 2016 10:55 AM
To: libfabric-users at lists.openfabrics.org
Subject: [libfabric-users] Using the verbs provider with manual address exchange

Hi,

I'm new to OFI, and I am writing a simple hello program in IB.

Looking at the example in fabtests, rc_pingpong.c uses RDMACM to
connect to peers. How do I do the address exchange myself ?

I tried to write something with AVs based on the tutorial/starting
guide, but maybe it is only for unconnected endpoints. Using AVs I can
get something working with sockets (though without the CQs) but not
with the verbs provider (fi_av_open is unimplemented with FI_MSG, and
with FI_RDM I can't get a verbs fabric object).

fi_info gives me:
provider: verbs
    fabric: IB-0x80fe
    domain: mlx4_0
    version: 1.0
    type: FI_EP_MSG
    protocol: FI_PROTO_RDMA_CM_IB_RC
provider: verbs
    fabric: IB-0x80fe
    domain: mlx4_0-rdm
    version: 1.0
    type: FI_EP_RDM
    protocol: FI_PROTO_IB_RDM

Thanks for some clarifications,
Piotr


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/libfabric-users/attachments/20160901/885debf2/attachment.html>