[libfabric-users] Multi process on same domain

[Xiong, Jianxin]

Libfabric "domain" is a per-process concept, so it is meaningless to talk about libfabric domain cross multiple processes.

The "-d" argument in the example specify the domain name, which maps to NIC device name for verbs. It is totally fine to create multiple domains with the same name.

A Verbs port can always be shared by multiple processes. Each process has its own "user context" when the Verbs device/port is opened. Note that the verbs "port number" is part of the device identification. It's different from the port number used in TCP connections.

-Jianxin

From: Libfabric-users <libfabric-users-bounces at lists.openfabrics.org> On Behalf Of Niyaz Murshed
Sent: Friday, August 30, 2024 7:27 AM
To: libfabric-users at lists.openfabrics.org
Cc: nd <nd at arm.com>
Subject: [libfabric-users] Multi process on same domain

Hello,

How do messages pass from QP to QP when both server and client are using the same domain?
I can see RoCE packets in Wireshark, so it does go to the NIC.


Client :
fi_rma_bw -s   192.168.2.100  -e msg   -I 1  -S 1024 -p verbs  -w 0  -o writedata 192.168.2.100 -d mlx5_2

Server:
fi_rma_bw -s   192.168.2.100  -e msg   -I 1  -S 1024 -p verbs  -w 0  -o writedata -d mlx5_2

I am trying to understand how multiple process/thread can share the same port over Verbs.

Regards,
Niyaz
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/libfabric-users/attachments/20240830/296f2326/attachment-0001.htm>