[libfabric-users] Can only use one NIC port in libfabric 1.6.1

[Ilango, Arun]

Hi Jörn,

The verbs provider assigns separate domains for each device got from rdma_get_devices(). So if the NIC ports show up as separate devices, they would belong to separate domains. This had been the case even for 1.4.

Can you check the output of ibv_devinfo? How does the ports show up there?

Thanks,
Arun.

-----Original Message-----
From: Libfabric-users [mailto:libfabric-users-bounces at lists.openfabrics.org] On Behalf Of Jörn Schumacher
Sent: Tuesday, August 21, 2018 2:17 AM
To: libfabric-users at lists.openfabrics.org
Subject: [libfabric-users] Can only use one NIC port in libfabric 1.6.1

Dear libfabric developers,

I recently updated to libfabric 1.6.1 (from 1.4). It looks like in this release we can only use on port of our NIC (Mellanox ConnectX-5 with RoCE).

On the receiving side we listen for a RC. We monitor the event queue with a file descriptor + epoll. On one port of the NIC this works fine, but if the request comes in on the second port (on a different IP
subnet) this fails: we get an epoll notification, but then the subsequent fi_eq_sread(...) call yields FI_EAGAIN.

I open a single domain. This worked fine in the earlier libfabric. 
Reading the documentation a bit I understand that a domain is tied to a port. Does this mean I need to open multiple domains?


Thanks and best regards,
Jörn
_______________________________________________
Libfabric-users mailing list
Libfabric-users at lists.openfabrics.org
https://lists.openfabrics.org/mailman/listinfo/libfabric-users