[libfabric-users] Multi process on same domain

[Xiong, Jianxin]

In the most general case, a packet would take a round-trip travel on the wire: Tx engine --> wire --> switch or loopback cable --> wire --> Rx engine. However, most NICs have a loopback path between the TX and RX engines so that the packet doesn’t need to go out to the wire for local bound communication.

-Jianxin

From: Niyaz Murshed <Niyaz.Murshed at arm.com>
Sent: Friday, August 30, 2024 8:25 AM
To: Xiong, Jianxin <jianxin.xiong at intel.com>; libfabric-users at lists.openfabrics.org
Cc: nd <nd at arm.com>
Subject: Re: Multi process on same domain

Thank you Jianxin.
yes, I meant 1 NIC being shared by 2 processes talking to each other on verbs.
I was wondering how the data is being exchanged between the QPs created since both QPs are on the same NIC.

Another test I am planning to do is as below :

2 servers , with 1 NIC connected to each other.

Client on Server A writes to Server on B over the NIC
Client on Server B writes to Server on A over the same NIC.

Basically, sharing the same NIC to send and receive work requests. These are just to understand the message flows in verbs.

From: Xiong, Jianxin <jianxin.xiong at intel.com<mailto:jianxin.xiong at intel.com>>
Date: Friday, August 30, 2024 at 10:17 AM
To: Niyaz Murshed <Niyaz.Murshed at arm.com<mailto:Niyaz.Murshed at arm.com>>, libfabric-users at lists.openfabrics.org<mailto:libfabric-users at lists.openfabrics.org> <libfabric-users at lists.openfabrics.org<mailto:libfabric-users at lists.openfabrics.org>>
Cc: nd <nd at arm.com<mailto:nd at arm.com>>
Subject: RE: Multi process on same domain
Libfabric “domain” is a per-process concept, so it is meaningless to talk about libfabric domain cross multiple processes.

The “-d” argument in the example specify the domain name, which maps to NIC device name for verbs. It is totally fine to create multiple domains with the same name.

A Verbs port can always be shared by multiple processes. Each process has its own “user context” when the Verbs device/port is opened. Note that the verbs “port number” is part of the device identification. It’s different from the port number used in TCP connections.

-Jianxin

From: Libfabric-users <libfabric-users-bounces at lists.openfabrics.org<mailto:libfabric-users-bounces at lists.openfabrics.org>> On Behalf Of Niyaz Murshed
Sent: Friday, August 30, 2024 7:27 AM
To: libfabric-users at lists.openfabrics.org<mailto:libfabric-users at lists.openfabrics.org>
Cc: nd <nd at arm.com<mailto:nd at arm.com>>
Subject: [libfabric-users] Multi process on same domain

Hello,

How do messages pass from QP to QP when both server and client are using the same domain?
I can see RoCE packets in Wireshark, so it does go to the NIC.


Client :
fi_rma_bw -s   192.168.2.100  -e msg   -I 1  -S 1024 -p verbs  -w 0  -o writedata 192.168.2.100 -d mlx5_2

Server:
fi_rma_bw -s   192.168.2.100  -e msg   -I 1  -S 1024 -p verbs  -w 0  -o writedata -d mlx5_2

I am trying to understand how multiple process/thread can share the same port over Verbs.

Regards,
Niyaz
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/libfabric-users/attachments/20240830/13bcd54c/attachment-0001.htm>