[libfabric-users] psm3 for Mellanox NIC

Niyaz Murshed Niyaz.Murshed at arm.com
Wed Jul 3 08:38:21 PDT 2024


Is it mandatory to install “Ethernet Fabric Suite FS Package” to use PSM3 provider ? If yes. What is it providing ?


From: Libfabric-users <libfabric-users-bounces at lists.openfabrics.org> on behalf of Niyaz Murshed <Niyaz.Murshed at arm.com>
Date: Wednesday, July 3, 2024 at 9:51 AM
To: Łukasz Ślusarczyk <lukasz.slusarczyk at intel.com>, libfabric-users at lists.openfabrics.org <libfabric-users at lists.openfabrics.org>
Cc: nd <nd at arm.com>
Subject: Re: [libfabric-users] psm3 for Mellanox NIC
I tried the steps.. however, now even the Intel NICS stopped showing PSM3 ☹




niymur01 at spr1:/opt/libfabric/bin$ ./fi_info -p psm3

provider: psm3

    fabric: TCP-10.118.91.0/24

    domain: eno1

    version: 306.10

    type: FI_EP_RDM

    protocol: FI_PROTO_PSMX3

provider: psm3

    fabric: TCP-10.118.91.0/24

    domain: eno1

    version: 306.10

    type: FI_EP_RDM

    protocol: FI_PROTO_PSMX3

provider: psm3

    fabric: TCP-10.118.91.0/24

    domain: eno1

    version: 306.10

    type: FI_EP_RDM

    protocol: FI_PROTO_PSMX3



niymur01 at spr1:~$ ls /usr/src/ofa_kernel/x86_64/$(uname -r)/

compat  compat_base  compat_base_tree  compat_base_tree_version  compat.config  compat_version  configure  configure.mk.kernel  include  Module.symvers  ofed_scripts


root at spr1:/tmp/Intel-Basic-IB.UBUNTU2204-x86_64.11.6.0.0.231#<mailto:root at spr1:/tmp/Intel-Basic-IB.UBUNTU2204-x86_64.11.6.0.0.231> ./INSTALL -a
Installing All Intel Software
Determining what is installed on system...
dpkg-query: no packages found matching kmod-iefs-kernel-updates
dpkg-query: no packages found matching kmod-iefs-kernel-updates
-------------------------------------------------------------------------------
Preparing OFA 11.6.0.0.231 release for Install...
Checking OS Dependencies needed for builds...
devscripts is required to build Eth Module
ERROR - unable to perform builds due to need for additional OS rpms
Unable to Prepare Kernel Module for Install
-------------------------------------------------------------------------------
Installing Intel...
Enable Adjusts kernel ARP table size for large fabrics. (ARPTABLE_TUNING)? ->y
installing [iefsconfig-11.6.0.0-231.all]...
Selecting previously unselected package iefsconfig.
(Reading database ... 211272 files and directories currently installed.)
Preparing to unpack .../iefsconfig_11.6.0.0-231_all.deb ...
Unpacking iefsconfig (11.6.0.0-231) ...
Setting up iefsconfig (11.6.0.0-231) ...
Created symlink /etc/systemd/system/multi-user.target.wants/iefs.service → /lib/systemd/system/iefs.service.
Processing triggers for man-db (2.10.2-1) ...
-------------------------------------------------------------------------------
Installing Basic Tools 11.6.0.0.231 release...
installing [eth-tools-basic-11.6.0.0-231.amd64]...
Selecting previously unselected package eth-tools-basic.
(Reading database ... 211293 files and directories currently installed.)
Preparing to unpack .../eth-tools-basic_11.6.0.0-231_amd64.deb ...
Unpacking eth-tools-basic (11.6.0.0-231) ...
dpkg: eth-tools-basic: dependency problems, but configuring anyway as you requested:
 eth-tools-basic depends on expect; however:
  Package expect is not installed.
 eth-tools-basic depends on snmpd; however:
  Package snmpd is not installed.

Setting up eth-tools-basic (11.6.0.0-231) ...
Processing triggers for man-db (2.10.2-1) ...
-------------------------------------------------------------------------------
Installing PSM3 11.6.0.0.231 release...
installing [libpsm3-fi-11.6.0.0-231.amd64]...
Selecting previously unselected package libpsm3-fi.
(Reading database ... 211314 files and directories currently installed.)
Preparing to unpack .../libpsm3-fi_11.6.0.0-231_amd64.deb ...
Unpacking libpsm3-fi (11.6.0.0-231) ...
Setting up libpsm3-fi (11.6.0.0-231) ...
Processing triggers for libc-bin (2.35-0ubuntu3.8) ...
Processing triggers for man-db (2.10.2-1) ...
-------------------------------------------------------------------------------
Installing Kernel Module  release...
get_kernel_rpms_to_install use dkms for eth_module
Not Found: ./IntelEth-OFA_DELTA.UBUNTU2204-x86_64.11.6.0.0.231/DEBS/ubuntu-UB2204/iefs-kernel-updates-dkms 6.5.0-28-generic
Not Found: ./IntelEth-OFA_DELTA.UBUNTU2204-x86_64.11.6.0.0.231/DEBS/ubuntu-UB2204/iefs-kernel-updates-devel 6.5.0-28-generic
-------------------------------------------------------------------------------
Installing Eth RoCE 11.6.0.0.231 release...
Enable RoCE RDMA transport. (ROCE_ON)? ->y
Resource Limits Selector. (0-7) -> 5
dpkg-query: no packages found matching kmod-iefs-kernel-updates
Enabling autostart for SNMP
-------------------------------------------------------------------------------



Reinstalled Libfabric.
After that :


root at spr1:/home/niymur01/libfabric# /opt/libfabric/bin/fi_info -p psm3
fi_getinfo: -61 (No data available)



From: Libfabric-users <libfabric-users-bounces at lists.openfabrics.org> on behalf of Niyaz Murshed <Niyaz.Murshed at arm.com>
Date: Thursday, June 20, 2024 at 2:30 PM
To: Łukasz Ślusarczyk <lukasz.slusarczyk at intel.com>, libfabric-users at lists.openfabrics.org <libfabric-users at lists.openfabrics.org>
Cc: nd <nd at arm.com>
Subject: Re: [libfabric-users] psm3 for Mellanox NIC
Thank you so much for the reply.
I will try this.

From: Łukasz Ślusarczyk <lukasz.slusarczyk at intel.com>
Date: Thursday, June 20, 2024 at 12:57 AM
To: Niyaz Murshed <Niyaz.Murshed at arm.com>, libfabric-users at lists.openfabrics.org <libfabric-users at lists.openfabrics.org>
Cc: nd <nd at arm.com>
Subject: Re: [libfabric-users] psm3 for Mellanox NIC

Hi!

It is supported. There is Intel Infiniband Package: https://www.intel.com/content/www/us/en/download/768193/intel-basic-infiniband-package.html

In section 4.5 of document describing installation of Intel's PSM3 https://downloadmirror.intel.com/818023/632484_Intel_Ethernet_Fabric_Software_Install_Guide_v1.7.pdf there are described optional steps for packages which support Mellanox OFED.

Note that not all Linux distributions work with IB (e.g. Ubuntu does not). See section 1.11 of https://downloadmirror.intel.com/812204/N36141_Intel%20Ethernet%20Fabric%20Software%20Release%20Notes_11.5.1.2_v1.2.pdf for supported distributions.

Regards,

Łukasz
On 20.06.2024 03:08, Niyaz Murshed wrote:
Hello All,

Do PSM3 provider support Mellanox NIC?


Root#/libfabric# lshw -c net -businfo -numeric

Bus info          Device          Class          Description

============================================================

pci at 0000<https://lists.openfabrics.org/mailman/listinfo/libfabric-users>:01:00.0<mailto:pci at 0000<https://lists.openfabrics.org/mailman/listinfo/libfabric-users>:01:00.0>  eno1            network        I210 Gigabit Network Connection [8086:1533]



pci at 0000<https://lists.openfabrics.org/mailman/listinfo/libfabric-users>:40:00.0<mailto:pci at 0000<https://lists.openfabrics.org/mailman/listinfo/libfabric-users>:40:00.0>  ens2f0np0       network        MT2910 Family [ConnectX-7] [15B3:1021] (Mellanox Technologies MT2910 Family [ConnectX-7])



When I run fi_info, psm3 is only enabled on for the intel nic



root#fi_info -p psm3

provider: psm3

    fabric: TCP-10.118.91.0/24

    domain: eno1

    version: 306.10

    type: FI_EP_RDM

    protocol: FI_PROTO_PSMX3

provider: psm3

    fabric: TCP-10.118.91.0/24

    domain: eno1

    version: 306.10

    type: FI_EP_RDM

    protocol: FI_PROTO_PSMX3

provider: psm3

    fabric: TCP-10.118.91.0/24

    domain: eno1

    version: 306.10

    type: FI_EP_RDM

    protocol: FI_PROTO_PSMX3




root # ibv_devinfo
hca_id:     mlx5_0
      transport:              InfiniBand (0)
      fw_ver:                       28.39.3004
      node_guid:              946d:ae03:007d:a230
      sys_image_guid:              946d:ae03:007d:a230
      vendor_id:              0x02c9
      vendor_part_id:              4129
      hw_ver:                       0x0
      board_id:               MT_0000000834
      phys_port_cnt:               1
            port: 1
                  state:                  PORT_ACTIVE (4)
                  max_mtu:          4096 (5)
                  active_mtu:       1024 (3)
                  sm_lid:                 0
                  port_lid:         0
                  port_lmc:         0x00
                  link_layer:       Ethernet




root # lscpu

Architecture:            x86_64

  CPU op-mode(s):        32-bit, 64-bit

Vendor ID:               GenuineIntel

  Model name:            Intel(R) Xeon(R) Platinum 8480+


Is this expected?



_______________________________________________

Libfabric-users mailing list

Libfabric-users at lists.openfabrics.org<mailto:Libfabric-users at lists.openfabrics.org>

https://lists.openfabrics.org/mailman/listinfo/libfabric-users

________________________________
Intel Technology Poland sp. z o.o.
ul. Słowackiego 173 | 80-298 Gdańsk | Sąd Rejonowy Gdańsk Północ | VII Wydział Gospodarczy Krajowego Rejestru Sądowego - KRS 101882 | NIP 957-07-52-316 | Kapitał zakładowy 200.000 PLN.
Spółka oświadcza, że posiada status dużego przedsiębiorcy w rozumieniu ustawy z dnia 8 marca 2013 r. o przeciwdziałaniu nadmiernym opóźnieniom w transakcjach handlowych.

Ta wiadomość wraz z załącznikami jest przeznaczona dla określonego adresata i może zawierać informacje poufne. W razie przypadkowego otrzymania tej wiadomości, prosimy o powiadomienie nadawcy oraz trwałe jej usunięcie; jakiekolwiek przeglądanie lub rozpowszechnianie jest zabronione.
This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). If you are not the intended recipient, please contact the sender and delete all copies; any review or distribution by others is strictly prohibited.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/libfabric-users/attachments/20240703/6d41e8e0/attachment-0001.htm>


More information about the Libfabric-users mailing list