[libfabric-users] Cannot establish a connection with verbs; ofi_rxm

[Hubert Hirtz]

Hi,

I am trying to make an application developed with the UDP provider
(using datagrams) work on InfiniBand hardware (with verbs and RxM).

The application does the following:

1. Initialize libfabric structures (EQ, CQ, AV, domain, provider) this way:

    eq_attr.size = 64;
    cq_attr.format = FI_CQ_FORMAT_MSG;
    cq_attr.size = 64;
    av_attr.flags = FI_EVENT;
    av_attr.type = FI_AV_TABLE;

    hints = fi_allocinfo();
    if (!hints)
        return -FI_ENOMEM;
    hints->ep_attr->type = FI_EP_RDM;
    hints->caps = FI_MSG | FI_SOURCE;
    hints->mode = FI_CONTEXT;
    fi_getinfo(FI_VERSION(1, 5), 0, port, port?FI_SOURCE:0, hints, &pv);

2. The server calls `fi_recv` and waits for a completion,

3. The client sends its socket address in a message to the server and
waits for a completion.

However, it seems the client-side `fi_cq_read` is always returning
`FI_EAGAIN`.  Also the `FI_CONTEXT` mode doesn't appear in the output of
`fi_tostr(pv, FI_TYPE_INFO)`.

The logs[0] show at line #178-187 that no verbs provider have been
found, however `fi_getinfo` still returns a valid verb provider.  I am
confused, could you please shed some light on this?

Cheers,
Hubert Hirtz

[0] https://paste.sr.ht/~taiite/3c575bd291301366cc7a6ce3474749c3a1ca420b