[libfabric-users] psm3 for Mellanox NIC
Niyaz Murshed
Niyaz.Murshed at arm.com
Wed Jul 3 07:49:46 PDT 2024
I tried the steps.. however, now even the Intel NICS stopped showing PSM3 ☹
niymur01 at spr1:/opt/libfabric/bin$ ./fi_info -p psm3
provider: psm3
fabric: TCP-10.118.91.0/24
domain: eno1
version: 306.10
type: FI_EP_RDM
protocol: FI_PROTO_PSMX3
provider: psm3
fabric: TCP-10.118.91.0/24
domain: eno1
version: 306.10
type: FI_EP_RDM
protocol: FI_PROTO_PSMX3
provider: psm3
fabric: TCP-10.118.91.0/24
domain: eno1
version: 306.10
type: FI_EP_RDM
protocol: FI_PROTO_PSMX3
niymur01 at spr1:~$ ls /usr/src/ofa_kernel/x86_64/$(uname -r)/
compat compat_base compat_base_tree compat_base_tree_version compat.config compat_version configure configure.mk.kernel include Module.symvers ofed_scripts
root at spr1:/tmp/Intel-Basic-IB.UBUNTU2204-x86_64.11.6.0.0.231#<mailto:root at spr1:/tmp/Intel-Basic-IB.UBUNTU2204-x86_64.11.6.0.0.231#> ./INSTALL -a
Installing All Intel Software
Determining what is installed on system...
dpkg-query: no packages found matching kmod-iefs-kernel-updates
dpkg-query: no packages found matching kmod-iefs-kernel-updates
-------------------------------------------------------------------------------
Preparing OFA 11.6.0.0.231 release for Install...
Checking OS Dependencies needed for builds...
devscripts is required to build Eth Module
ERROR - unable to perform builds due to need for additional OS rpms
Unable to Prepare Kernel Module for Install
-------------------------------------------------------------------------------
Installing Intel...
Enable Adjusts kernel ARP table size for large fabrics. (ARPTABLE_TUNING)? ->y
installing [iefsconfig-11.6.0.0-231.all]...
Selecting previously unselected package iefsconfig.
(Reading database ... 211272 files and directories currently installed.)
Preparing to unpack .../iefsconfig_11.6.0.0-231_all.deb ...
Unpacking iefsconfig (11.6.0.0-231) ...
Setting up iefsconfig (11.6.0.0-231) ...
Created symlink /etc/systemd/system/multi-user.target.wants/iefs.service → /lib/systemd/system/iefs.service.
Processing triggers for man-db (2.10.2-1) ...
-------------------------------------------------------------------------------
Installing Basic Tools 11.6.0.0.231 release...
installing [eth-tools-basic-11.6.0.0-231.amd64]...
Selecting previously unselected package eth-tools-basic.
(Reading database ... 211293 files and directories currently installed.)
Preparing to unpack .../eth-tools-basic_11.6.0.0-231_amd64.deb ...
Unpacking eth-tools-basic (11.6.0.0-231) ...
dpkg: eth-tools-basic: dependency problems, but configuring anyway as you requested:
eth-tools-basic depends on expect; however:
Package expect is not installed.
eth-tools-basic depends on snmpd; however:
Package snmpd is not installed.
Setting up eth-tools-basic (11.6.0.0-231) ...
Processing triggers for man-db (2.10.2-1) ...
-------------------------------------------------------------------------------
Installing PSM3 11.6.0.0.231 release...
installing [libpsm3-fi-11.6.0.0-231.amd64]...
Selecting previously unselected package libpsm3-fi.
(Reading database ... 211314 files and directories currently installed.)
Preparing to unpack .../libpsm3-fi_11.6.0.0-231_amd64.deb ...
Unpacking libpsm3-fi (11.6.0.0-231) ...
Setting up libpsm3-fi (11.6.0.0-231) ...
Processing triggers for libc-bin (2.35-0ubuntu3.8) ...
Processing triggers for man-db (2.10.2-1) ...
-------------------------------------------------------------------------------
Installing Kernel Module release...
get_kernel_rpms_to_install use dkms for eth_module
Not Found: ./IntelEth-OFA_DELTA.UBUNTU2204-x86_64.11.6.0.0.231/DEBS/ubuntu-UB2204/iefs-kernel-updates-dkms 6.5.0-28-generic
Not Found: ./IntelEth-OFA_DELTA.UBUNTU2204-x86_64.11.6.0.0.231/DEBS/ubuntu-UB2204/iefs-kernel-updates-devel 6.5.0-28-generic
-------------------------------------------------------------------------------
Installing Eth RoCE 11.6.0.0.231 release...
Enable RoCE RDMA transport. (ROCE_ON)? ->y
Resource Limits Selector. (0-7) -> 5
dpkg-query: no packages found matching kmod-iefs-kernel-updates
Enabling autostart for SNMP
-------------------------------------------------------------------------------
Reinstalled Libfabric.
After that :
root at spr1:/home/niymur01/libfabric# /opt/libfabric/bin/fi_info -p psm3
fi_getinfo: -61 (No data available)
From: Libfabric-users <libfabric-users-bounces at lists.openfabrics.org> on behalf of Niyaz Murshed <Niyaz.Murshed at arm.com>
Date: Thursday, June 20, 2024 at 2:30 PM
To: Łukasz Ślusarczyk <lukasz.slusarczyk at intel.com>, libfabric-users at lists.openfabrics.org <libfabric-users at lists.openfabrics.org>
Cc: nd <nd at arm.com>
Subject: Re: [libfabric-users] psm3 for Mellanox NIC
Thank you so much for the reply.
I will try this.
From: Łukasz Ślusarczyk <lukasz.slusarczyk at intel.com>
Date: Thursday, June 20, 2024 at 12:57 AM
To: Niyaz Murshed <Niyaz.Murshed at arm.com>, libfabric-users at lists.openfabrics.org <libfabric-users at lists.openfabrics.org>
Cc: nd <nd at arm.com>
Subject: Re: [libfabric-users] psm3 for Mellanox NIC
Hi!
It is supported. There is Intel Infiniband Package: https://www.intel.com/content/www/us/en/download/768193/intel-basic-infiniband-package.html
In section 4.5 of document describing installation of Intel's PSM3 https://downloadmirror.intel.com/818023/632484_Intel_Ethernet_Fabric_Software_Install_Guide_v1.7.pdf there are described optional steps for packages which support Mellanox OFED.
Note that not all Linux distributions work with IB (e.g. Ubuntu does not). See section 1.11 of https://downloadmirror.intel.com/812204/N36141_Intel%20Ethernet%20Fabric%20Software%20Release%20Notes_11.5.1.2_v1.2.pdf for supported distributions.
Regards,
Łukasz
On 20.06.2024 03:08, Niyaz Murshed wrote:
Hello All,
Do PSM3 provider support Mellanox NIC?
Root#/libfabric# lshw -c net -businfo -numeric
Bus info Device Class Description
============================================================
pci at 0000<https://lists.openfabrics.org/mailman/listinfo/libfabric-users>:01:00.0<mailto:pci at 0000<https://lists.openfabrics.org/mailman/listinfo/libfabric-users>:01:00.0> eno1 network I210 Gigabit Network Connection [8086:1533]
pci at 0000<https://lists.openfabrics.org/mailman/listinfo/libfabric-users>:40:00.0<mailto:pci at 0000<https://lists.openfabrics.org/mailman/listinfo/libfabric-users>:40:00.0> ens2f0np0 network MT2910 Family [ConnectX-7] [15B3:1021] (Mellanox Technologies MT2910 Family [ConnectX-7])
When I run fi_info, psm3 is only enabled on for the intel nic
root#fi_info -p psm3
provider: psm3
fabric: TCP-10.118.91.0/24
domain: eno1
version: 306.10
type: FI_EP_RDM
protocol: FI_PROTO_PSMX3
provider: psm3
fabric: TCP-10.118.91.0/24
domain: eno1
version: 306.10
type: FI_EP_RDM
protocol: FI_PROTO_PSMX3
provider: psm3
fabric: TCP-10.118.91.0/24
domain: eno1
version: 306.10
type: FI_EP_RDM
protocol: FI_PROTO_PSMX3
root # ibv_devinfo
hca_id: mlx5_0
transport: InfiniBand (0)
fw_ver: 28.39.3004
node_guid: 946d:ae03:007d:a230
sys_image_guid: 946d:ae03:007d:a230
vendor_id: 0x02c9
vendor_part_id: 4129
hw_ver: 0x0
board_id: MT_0000000834
phys_port_cnt: 1
port: 1
state: PORT_ACTIVE (4)
max_mtu: 4096 (5)
active_mtu: 1024 (3)
sm_lid: 0
port_lid: 0
port_lmc: 0x00
link_layer: Ethernet
root # lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Vendor ID: GenuineIntel
Model name: Intel(R) Xeon(R) Platinum 8480+
Is this expected?
_______________________________________________
Libfabric-users mailing list
Libfabric-users at lists.openfabrics.org<mailto:Libfabric-users at lists.openfabrics.org>
https://lists.openfabrics.org/mailman/listinfo/libfabric-users
________________________________
Intel Technology Poland sp. z o.o.
ul. Słowackiego 173 | 80-298 Gdańsk | Sąd Rejonowy Gdańsk Północ | VII Wydział Gospodarczy Krajowego Rejestru Sądowego - KRS 101882 | NIP 957-07-52-316 | Kapitał zakładowy 200.000 PLN.
Spółka oświadcza, że posiada status dużego przedsiębiorcy w rozumieniu ustawy z dnia 8 marca 2013 r. o przeciwdziałaniu nadmiernym opóźnieniom w transakcjach handlowych.
Ta wiadomość wraz z załącznikami jest przeznaczona dla określonego adresata i może zawierać informacje poufne. W razie przypadkowego otrzymania tej wiadomości, prosimy o powiadomienie nadawcy oraz trwałe jej usunięcie; jakiekolwiek przeglądanie lub rozpowszechnianie jest zabronione.
This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). If you are not the intended recipient, please contact the sender and delete all copies; any review or distribution by others is strictly prohibited.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/libfabric-users/attachments/20240703/ac1f8622/attachment-0001.htm>
More information about the Libfabric-users
mailing list