<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<div style="margin: 0px; font-size: 12pt; color: black; background-color: white;">
Hello,</div>
<div style="margin: 0px; font-size: 12pt; color: black; background-color: white;">
<br>
<div style="margin:0px">I am testing the newly upgraded OFED (5.1-0.6.6) and corresponding OpenMPI (4.0.2, 4.0.4).</div>
<div style="margin:0px"><br>
</div>
<div style="margin:0px">I don't know for what reason, but I get a communication error. (There is no error in the combination of OFED(4.6-1.0.1) & OpenMPI(4.0.2))</div>
<div style="margin:0px"><br>
</div>
<div style="margin:0px"></div>
When communicating between compute nodes(inter-nodes), if the size of send/recv messages exceeds 65535, the following error occurs.</div>
<div style="margin: 0px; font-size: 12pt; color: black; background-color: white;">
This does not happen when using one compute node.
<div style="margin:0px"><br>
</div>
<div style="margin:0px">If there are any points worth checking, it would be appreciated if you could tell us even a trivial thing.<br>
</div>
<div style="margin:0px"><br>
</div>
<div style="margin:0px"><br>
</div>
<div style="margin:0px">
<div style="margin: 0px; background-color: white; color: black;">Best Regards,</div>
<div style="margin: 0px; background-color: white; color: black;">Kihang<br>
</div>
<div style="margin: 0px; background-color: white; color: black;"><br>
</div>
</div>
<div style="margin: 0px; background-color: white; color: black;"><br>
</div>
<div style="margin:0px"><br>
</div>
<div style="margin:0px"><b>Part of the error message:<br>
</b></div>
<div style="margin:0px"><br>
</div>
<div style="margin:0px">[pduru18:351568:0:351568] ib_mlx5_log.c:143 Transport retry count exceeded on mlx5_2:1/RoCE (synd 0x15 vend 0x81 hw_synd 0/0)
<div style="margin:0px">[pduru18:351568:0:351568] ib_mlx5_log.c:143 RC QP 0x139d4 wqe[0]: RDMA_READ s-- [rva 0x2b9827e90a40 rkey 0x182ab] [va 0x2b270e05ca00 len 219136 lkey 0x3c2b]</div>
<div style="margin:0px">[pduru18:351565:0:351565] ib_mlx5_log.c:143 Transport retry count exceeded on mlx5_2:1/RoCE (synd 0x15 vend 0x81 hw_synd 0/0)</div>
<div style="margin:0px">[pduru18:351565:0:351565] ib_mlx5_log.c:143 RC QP 0x139d3 wqe[0]: RDMA_READ s-- [rva 0x2ac9d73be980 rkey 0x8b395] [va 0x2b464c51bc00 len 223232 lkey 0x5e4b]</div>
<div style="margin:0px">[pduru18:351571:0:351571] ib_mlx5_log.c:143 Transport retry count exceeded on mlx5_2:1/RoCE (synd 0x15 vend 0x81 hw_synd 0/0)</div>
<div style="margin:0px">[pduru18:351571:0:351571] ib_mlx5_log.c:143 RC QP 0x139d2 wqe[0]: RDMA_READ s-- [rva 0x2b0072dd1980 rkey 0x55fea] [va 0x2b70590d8c00 len 223232 lkey 0x715b]</div>
</div>
<div style="margin:0px"><br>
</div>
<b>Executable file error message:<br>
</b></div>
<div style="margin: 0px; font-size: 15px; font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
<div style="margin: 0px; font-size: 12pt; font-family: Calibri, Arial, Helvetica, sans-serif; color: black;">
<br>
</div>
<div style="margin: 0px; font-size: 12pt; font-family: Calibri, Arial, Helvetica, sans-serif; color: black;">
<div style="margin:0px">==== backtrace (tid: 351569) ====</div>
<div style="margin:0px"> 0 0x000000000004ed85 ucs_debug_print_backtrace() ???:0</div>
<div style="margin:0px"> 1 0x000000000001f9c2 uct_ib_mlx5_completion_with_err() ???:0</div>
<div style="margin:0px"> 2 0x000000000002e736 uct_rc_mlx5_iface_is_reachable() ???:0</div>
<div style="margin:0px"> 3 0x0000000000030481 uct_rc_mlx5_iface_progress() ???:0</div>
<div style="margin:0px"> 4 0x0000000000022f3a ucp_worker_progress() ???:0</div>
<div style="margin:0px"> 5 0x0000000000038574 opal_progress() /export/home/nwp/OFED_TEST/KMALIB/src/openmpi/openmpi-4.0.4/opal/runtime/opal_progress.c:231</div>
<div style="margin:0px"> 6 0x00000000000569f7 ompi_request_wait_completion() /export/home/nwp/OFED_TEST/KMALIB/src/openmpi/openmpi-4.0.4/ompi/../ompi/request/request.h:415</div>
<div style="margin:0px"> 7 0x00000000000569f7 ompi_request_default_wait() /export/home/nwp/OFED_TEST/KMALIB/src/openmpi/openmpi-4.0.4/ompi/request/req_wait.c:42<br>
</div>
<div style="margin:0px"> 8 0x0000000000084772 PMPI_Wait() /export/home/nwp/OFED_TEST/KMALIB/src/openmpi/openmpi-4.0.4/ompi/mpi/c/profile/pwait.c:74</div>
<div style="margin:0px"> 9 0x000000000005b26f ompi_wait_f() /export/home/nwp/OFED_TEST/KMALIB/src/openmpi/openmpi-4.0.4/ompi/mpi/fortran/mpif-h/profile/pwait_f.c:76<br>
</div>
<div style="margin:0px">10 0x00000000005b1642 swap3d_() ???:0</div>
<div style="margin:0px">11 0x00000000004a6eb4 hdiff_() ???:0</div>
<div style="margin:0px">12 0x000000000046bf81 sciproc_() ???:0</div>
<div style="margin:0px">13 0x0000000000462418 MAIN__() ???:0<br>
</div>
<div style="margin:0px">14 0x000000000040bfde main() ???:0</div>
<div style="margin:0px">15 0x00000000000223d5 __libc_start_main() ???:0</div>
16 0x000000000040bee9 _start() ???:0<br>
</div>
<div style="margin: 0px; font-size: 12pt; font-family: Calibri, Arial, Helvetica, sans-serif; color: black;">
<br>
</div>
<div style="margin: 0px; font-size: 12pt; font-family: Calibri, Arial, Helvetica, sans-serif; color: black;">
<br>
</div>
</div>
</div>
<div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<b>> ucx_info -v</b>
<div># UCT version=1.9.0 revision 1d0a420</div>
<div># configured with: --build=x86_64-redhat-linux-gnu --host=x86_64-redhat-linux-gnu --program-prefix= --disable-dependency-tracking --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc --datadir=/usr/share --includedir=/usr/include
--libdir=/usr/lib64 --libexecdir=/usr/libexec --localstatedir=/var --sharedstatedir=/var/lib --mandir=/usr/share/man --infodir=/usr/share/info --disable-optimizations --disable-logging --disable-debug --disable-assertions --enable-mt --disable-params-check
--without-java --enable-cma --without-cuda --without-gdrcopy --with-verbs --without-cm --with-knem --with-rdmacm --without-rocm --without-xpmem --without-ugni</div>
<div><br>
</div>
<div><b>> cat /etc/redhat-release</b></div>
<div>CentOS Linux release 7.6.1810 (Core)</div>
<div><br>
</div>
<div><b>> uname -a</b></div>
<div>Linux boot2 3.10.0-957.el7.x86_64 #1 SMP Thu Nov 8 23:39:32 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux</div>
<div><br>
</div>
<div><b>> ofed_info -s</b></div>
<div>MLNX_OFED_LINUX-5.1-0.6.6.0:</div>
<div><br>
</div>
<div><b>> ibstat</b></div>
<div>CA 'mlx5_0'</div>
<div> CA type: MT4123</div>
<div> Number of ports: 1</div>
<div> Firmware version: 20.28.1002</div>
<div> Hardware version: 0</div>
<div> Node GUID: 0xb8599f0300b84da6</div>
<div> System image GUID: 0xb8599f0300b84da6</div>
<div> Port 1:</div>
<div> State: Active</div>
<div> Physical state: LinkUp</div>
<div> Rate: 100</div>
<div> Base lid: 4</div>
<div> LMC: 0</div>
<div> SM lid: 4</div>
<div> Capability mask: 0x2651e84a</div>
<div> Port GUID: 0xb8599f0300b84da6</div>
<div> Link layer: InfiniBand</div>
<div><br>
</div>
<div><b>> ibv_devinfo -v</b></div>
<div>hca_id: mlx5_0</div>
<div> transport: InfiniBand (0)</div>
<div> fw_ver: 20.28.1002</div>
<div> node_guid: b859:9f03:00b8:4da6</div>
<div> sys_image_guid: b859:9f03:00b8:4da6</div>
<div> vendor_id: 0x02c9</div>
<div> vendor_part_id: 4123</div>
<div> hw_ver: 0x0</div>
<div> board_id: LNV0000000016</div>
<div> phys_port_cnt: 1</div>
<div> max_mr_size: 0xffffffffffffffff</div>
<div> page_size_cap: 0xfffffffffffff000</div>
<div> max_qp: 262144</div>
<div> max_qp_wr: 32768</div>
<div> device_cap_flags: 0xe97e1c36</div>
<div> BAD_PKEY_CNTR</div>
<div> BAD_QKEY_CNTR</div>
<div> AUTO_PATH_MIG</div>
<div> CHANGE_PHY_PORT</div>
<div> PORT_ACTIVE_EVENT</div>
<div> SYS_IMAGE_GUID</div>
<div> RC_RNR_NAK_GEN</div>
<div> MEM_WINDOW</div>
<div> UD_IP_CSUM</div>
<div> XRC</div>
<div> MEM_MGT_EXTENSIONS</div>
<div> MEM_WINDOW_TYPE_2B</div>
<div> MANAGED_FLOW_STEERING</div>
<div> Unknown flags: 0xC8480000</div>
<div> max_sge: 30</div>
<div> max_sge_rd: 30</div>
<div> max_cq: 16777216</div>
<div> max_cqe: 4194303</div>
<div> max_mr: 16777216</div>
<div> max_pd: 16777216</div>
<div> max_qp_rd_atom: 16</div>
<div> max_ee_rd_atom: 0</div>
<div> max_res_rd_atom: 4194304</div>
<div> max_qp_init_rd_atom: 16</div>
<div> max_ee_init_rd_atom: 0</div>
<div> atomic_cap: ATOMIC_HCA (1)</div>
<div> max_ee: 0</div>
<div> max_rdd: 0</div>
<div> max_mw: 16777216</div>
<div> max_raw_ipv6_qp: 0</div>
<div> max_raw_ethy_qp: 0</div>
<div> max_mcast_grp: 2097152</div>
<div> max_mcast_qp_attach: 240</div>
<div> max_total_mcast_qp_attach: 503316480</div>
<div> max_ah: 2147483647</div>
<div> max_fmr: 0</div>
<div> max_srq: 8388608</div>
<div> max_srq_wr: 32767</div>
<div> max_srq_sge: 31</div>
<div> max_pkeys: 128</div>
<div> local_ca_ack_delay: 16</div>
<div> general_odp_caps:</div>
<div> ODP_SUPPORT</div>
<div> ODP_SUPPORT_IMPLICIT</div>
<div> rc_odp_caps:</div>
<div> SUPPORT_SEND</div>
<div> SUPPORT_RECV</div>
<div> SUPPORT_WRITE</div>
<div> SUPPORT_READ</div>
<div> SUPPORT_SRQ</div>
<div> uc_odp_caps:</div>
<div> NO SUPPORT</div>
<div> ud_odp_caps:</div>
<div> SUPPORT_SEND</div>
<div> xrc_odp_caps:</div>
<div> SUPPORT_SEND</div>
<div> SUPPORT_WRITE</div>
<div> SUPPORT_READ</div>
<div> SUPPORT_SRQ</div>
<div> completion timestamp_mask: 0x7fffffffffffffff</div>
<div> hca_core_clock: 156250kHZ</div>
<div> device_cap_flags_ex: 0x30000051E97E1C36</div>
<div> PCI_WRITE_END_PADDING</div>
<div> Unknown flags: 0x3000004100000000</div>
<div> tso_caps:</div>
<div> max_tso: 0</div>
<div> rss_caps:</div>
<div> max_rwq_indirection_tables: 0</div>
<div> max_rwq_indirection_table_size: 0</div>
<div> rx_hash_function: 0x0</div>
<div> rx_hash_fields_mask: 0x0</div>
<div> max_wq_type_rq: 0</div>
<div> packet_pacing_caps:</div>
<div> qp_rate_limit_min: 0kbps</div>
<div> qp_rate_limit_max: 0kbps</div>
<div> max_rndv_hdr_size: 64</div>
<div> max_num_tags: 127</div>
<div> max_ops: 32768</div>
<div> max_sge: 1</div>
<div> flags:</div>
<div> IBV_TM_CAP_RC</div>
<div><br>
</div>
<div> cq moderation caps:</div>
<div> max_cq_count: 65535</div>
<div> max_cq_period: 4095 us</div>
<div><br>
</div>
<div> maximum available device memory: 262144Bytes</div>
<div><br>
</div>
<div> port: 1</div>
<div> state: PORT_ACTIVE (4)</div>
<div> max_mtu: 4096 (5)</div>
<div> active_mtu: 4096 (5)</div>
<div> sm_lid: 4</div>
<div> port_lid: 4</div>
<div> port_lmc: 0x00</div>
<div> link_layer: InfiniBand</div>
<div> max_msg_sz: 0x40000000</div>
<div> port_cap_flags: 0x2251e84a</div>
<div> port_cap_flags2: 0x0032</div>
<div> max_vl_num: 4 (3)</div>
<div> bad_pkey_cntr: 0x0</div>
<div> qkey_viol_cntr: 0x0</div>
<div> sm_sl: 0</div>
<div> pkey_tbl_len: 128</div>
<div> gid_tbl_len: 8</div>
<div> subnet_timeout: 18</div>
<div> init_type_reply: 0</div>
<div> active_width: 2X (16)</div>
<div> active_speed: 50.0 Gbps (64)</div>
<div> phys_state: LINK_UP (5)</div>
<div> GID[ 0]: fe80:0000:0000:0000:b859:9f03:00b8:4da6</div>
<div><br>
</div>
<div><span style=""><b>> ompi_info</b></span></div>
<div> Package: Open MPI root@boot2 Distribution</div>
<div> Open MPI: 4.0.2</div>
<div> Open MPI repo revision: v4.0.2</div>
<div> Open MPI release date: Oct 07, 2019</div>
<div> Open RTE: 4.0.2</div>
<div> Open RTE repo revision: v4.0.2</div>
<div> Open RTE release date: Oct 07, 2019</div>
<div> OPAL: 4.0.2</div>
<div> OPAL repo revision: v4.0.2</div>
<div> OPAL release date: Oct 07, 2019</div>
<div> MPI API: 3.1.0</div>
<div> Ident string: 4.0.2</div>
<div> Prefix: /d1/home/nwp/OFED_TEST/KMALIB/apps/openmpi/4.0.2_ofed</div>
<div> Configured architecture: x86_64-unknown-linux-gnu</div>
<div> Configure host: boot2</div>
<div> Configured by: root</div>
<div> Configured on: Wed Dec 16 06:21:49 UTC 2020</div>
<div> Configure host: boot2</div>
<div> Configure command line: 'CC=icc' 'CFLAGS=-m64' 'FC=ifort' 'FCFLAGS=-m64'</div>
<div> '--prefix=/d1/home/nwp/OFED_TEST/KMALIB/apps/openmpi/4.0.2_ofed'</div>
<div> '--with-platform=mellanox/optimized'</div>
<div> '--with-mxm=/opt/mellanox/mxm'</div>
<div> '--with-knem=/opt/knem-1.1.4.90mlnx1/'</div>
<div> '--with-zlib=/opt/kma/kma_lib/apps/zlib/1.2.11/'</div>
<div> '--with-zlib-libdir=/opt/kma/kma_lib/apps/zlib/1.2.11/lib'</div>
<div> '--with-lsf=/opt/ibm/lsfsuite/dcomp/lsf/10.1'</div>
<div> '--with-lsf-libdir=/opt/ibm/lsfsuite/lsf/10.1/linux2.6-glibc2.3-x86_64/lib/'</div>
<div> Built by: root</div>
<div> Built on: Wed Dec 16 06:29:41 UTC 2020</div>
<div> Built host: boot2</div>
<div> C bindings: yes</div>
<div> C++ bindings: no</div>
<div> Fort mpif.h: yes (all)</div>
<div> Fort use mpi: yes (full: ignore TKR)</div>
<div> Fort use mpi size: deprecated-ompi-info-value</div>
<div> Fort use mpi_f08: yes</div>
<div> Fort mpi_f08 compliance: The mpi_f08 module is available, but due to</div>
<div> limitations in the ifort compiler and/or Open MPI,</div>
<div> does not support the following: array subsections,</div>
<div> direct passthru (where possible) to underlying Open</div>
<div> MPI's C functionality</div>
<div> Fort mpi_f08 subarrays: no</div>
<div> Java bindings: no</div>
<div> Wrapper compiler rpath: runpath</div>
<div> C compiler: icc</div>
<div> C compiler absolute: /d1/home/nwp/OFED_TEST/KMALIB/apps/intel/20.2_ofed/compilers_and_libraries_2020/linux/bin/intel64/icc</div>
<div> C compiler family name: INTEL</div>
<div> C compiler version: 1910.20200623</div>
<div> C++ compiler: g++</div>
<div> C++ compiler absolute: /opt/kma/kma_lib/apps/gcc/7.5.0/bin/g++</div>
<div> Fort compiler: ifort</div>
<div> Fort compiler abs: /d1/home/nwp/OFED_TEST/KMALIB/apps/intel/20.2_ofed/compilers_and_libraries_2020/linux/bin/intel64/ifort</div>
<div> Fort ignore TKR: yes (!DEC$ ATTRIBUTES NO_ARG_CHECK ::)</div>
<div> Fort 08 assumed shape: yes</div>
<div> Fort optional args: yes</div>
<div> Fort INTERFACE: yes</div>
<div> Fort ISO_FORTRAN_ENV: yes</div>
<div> Fort STORAGE_SIZE: yes</div>
<div> Fort BIND(C) (all): yes</div>
<div> Fort ISO_C_BINDING: yes</div>
<div> Fort SUBROUTINE BIND(C): yes</div>
<div> Fort TYPE,BIND(C): yes</div>
<div> Fort T,BIND(C,name="a"): yes</div>
<div> Fort PRIVATE: yes</div>
<div> Fort PROTECTED: yes</div>
<div> Fort ABSTRACT: yes</div>
<div> Fort ASYNCHRONOUS: yes</div>
<div> Fort PROCEDURE: yes</div>
<div> Fort USE...ONLY: yes</div>
<div> Fort C_FUNLOC: yes</div>
<div> Fort f08 using wrappers: yes</div>
<div> Fort MPI_SIZEOF: yes</div>
<div> C profiling: yes</div>
<div> C++ profiling: no</div>
<div> Fort mpif.h profiling: yes</div>
<div> Fort use mpi profiling: yes</div>
<div> Fort use mpi_f08 prof: yes</div>
<div> C++ exceptions: no</div>
<div> Thread support: posix (MPI_THREAD_MULTIPLE: yes, OPAL support: yes,</div>
<div> OMPI progress: no, ORTE progress: yes, Event lib:</div>
<div> yes)</div>
<div> Sparse Groups: no</div>
<div> Internal debug support: no</div>
<div> MPI interface warnings: yes</div>
<div> MPI parameter check: never</div>
<div>Memory profiling support: no</div>
<div>Memory debugging support: no</div>
<div> dl support: yes</div>
<div> Heterogeneous support: no</div>
<div> mpirun default --prefix: yes</div>
<div> MPI_WTIME support: native</div>
<div> Symbol vis. support: yes</div>
<div> Host topology support: yes</div>
<div> IPv6 support: no</div>
<div> MPI1 compatibility: no</div>
<div> MPI extensions: affinity, cuda, pcollreq</div>
<div> FT Checkpoint support: no (checkpoint thread: no)</div>
<div> C/R Enabled Debugging: no</div>
<div> MPI_MAX_PROCESSOR_NAME: 256</div>
<div> MPI_MAX_ERROR_STRING: 256</div>
<div> MPI_MAX_OBJECT_NAME: 64</div>
<div> MPI_MAX_INFO_KEY: 36</div>
<div> MPI_MAX_INFO_VAL: 256</div>
<div> MPI_MAX_PORT_NAME: 1024</div>
<div> MPI_MAX_DATAREP_STRING: 128</div>
<div><br>
</div>
<div><b>> ofed_info</b></div>
<div>MLNX_OFED_LINUX-5.1-0.6.6.0 (OFED-5.1-0.6.6):</div>
<div>ar_mgr:</div>
<div>osm_plugins/ar_mgr/ar_mgr-1.0-0.2.MLNX20200630.g8577618.tar.gz</div>
<div><br>
</div>
<div>dpcp:</div>
<div>/sw/release/sw_acceleration/dpcp/dpcp-1.0.0-1.src.rpm</div>
<div><br>
</div>
<div>dump_pr:</div>
<div>osm_plugins/dump_pr//dump_pr-1.0-0.2.MLNX20200630.g8577618.tar.gz</div>
<div><br>
</div>
<div>fabric-collector:</div>
<div>fabric_collector//fabric-collector-1.1.0.MLNX20170103.89bb2aa.tar.gz</div>
<div><br>
</div>
<div>hcoll:</div>
<div>mlnx_ofed_hcol/hcoll-4.6.3125-1.src.rpm</div>
<div><br>
</div>
<div>ibdump:</div>
<div>https://github.com/Mellanox/ibdump master</div>
<div>commit 6355ebbd664cafb629edeadecd4096ac2a0304c3</div>
<div>ibsim:</div>
<div>mlnx_ofed_ibsim/ibsim-0.9.tar.gz</div>
<div><br>
</div>
<div>ibutils2:</div>
<div>ibutils2/ibutils2-2.1.1-0.126.MLNX20200721.gf95236b.tar.gz</div>
<div><br>
</div>
<div>iser:</div>
<div>mlnx_ofed/mlnx-ofa_kernel-4.0.git mlnx_ofed_5_1</div>
<div>commit c72091bb7f69243219dda60946342385c9766aa3</div>
<div><br>
</div>
<div>isert:</div>
<div>mlnx_ofed/mlnx-ofa_kernel-4.0.git mlnx_ofed_5_1</div>
<div>commit c72091bb7f69243219dda60946342385c9766aa3</div>
<div><br>
</div>
<div>kernel-mft:</div>
<div>mlnx_ofed_mft/kernel-mft-4.15.0-104.src.rpm</div>
<div><br>
</div>
<div>knem:</div>
<div>knem.git mellanox-master</div>
<div>commit 299ba51259c0947b71b762567bccf660513f8643</div>
<div>libpka:</div>
<div>mlnx_ofed_soc/libpka-1.0-1.gcc98895.src.rpm</div>
<div><br>
</div>
<div>libvma:</div>
<div>vma/source_rpms/libvma-9.1.1-0.src.rpm</div>
<div><br>
</div>
<div>mlnx-dpdk:</div>
<div>https://github.com/Mellanox/dpdk.org mlnx_dpdk_19.11_last_stable</div>
<div>commit c8732df963abf855edf2447a0b8d8543e7924ba9</div>
<div>mlnx-en:</div>
<div>mlnx_ofed/mlnx-ofa_kernel-4.0.git mlnx_ofed_5_1</div>
<div>commit c72091bb7f69243219dda60946342385c9766aa3</div>
<div><br>
</div>
<div>mlnx-ethtool:</div>
<div>mlnx_ofed/ethtool.git mlnx_ofed_5_1</div>
<div>commit a1f6f627af80b76b013b68ff57a3ae41ac7517f9</div>
<div>mlnx-iproute2:</div>
<div>mlnx_ofed/iproute2.git mlnx_ofed_5_1</div>
<div>commit 9a007c2d912ce52ad5e3e9c6a9bc9fb4d20fd52c</div>
<div>mlnx-nfsrdma:</div>
<div>mlnx_ofed/mlnx-ofa_kernel-4.0.git mlnx_ofed_5_1</div>
<div>commit c72091bb7f69243219dda60946342385c9766aa3</div>
<div><br>
</div>
<div>mlnx-nvme:</div>
<div>mlnx_ofed/mlnx-ofa_kernel-4.0.git mlnx_ofed_5_1</div>
<div>commit c72091bb7f69243219dda60946342385c9766aa3</div>
<div><br>
</div>
<div>mlnx-ofa_kernel:</div>
<div>mlnx_ofed/mlnx-ofa_kernel-4.0.git mlnx_ofed_5_1</div>
<div>commit c72091bb7f69243219dda60946342385c9766aa3</div>
<div><br>
</div>
<div>mlxbf-bootctl:</div>
<div>https://github.com/Mellanox/mlxbf-bootctl bluefield-rel/3.0</div>
<div>commit fda69b62ac4f2707a82da18f894b40120f686010</div>
<div>mpi-selector:</div>
<div>ofed-1.5.3-rpms/mpi-selector/mpi-selector-1.0.3-1.src.rpm</div>
<div><br>
</div>
<div>mpitests:</div>
<div>mlnx_ofed_mpitest/mpitests-3.2.20-5d20b49.src.rpm</div>
<div><br>
</div>
<div>mstflint:</div>
<div>mlnx_ofed_mstflint/mstflint-4.14.0-3.tar.gz</div>
<div><br>
</div>
<div>multiperf:</div>
<div>mlnx_ofed_multiperf/multiperf-3.0-0.14.g5f0fd0e.tar.gz</div>
<div><br>
</div>
<div>mxm:</div>
<div>/sw/release/mlnx_ofed/IBHPC/MLNX_OFED_LINUX-5.0-0.3.7/SRPMS/mxm-3.7.3112-1.50037.src.rpm</div>
<div><br>
</div>
<div>ofed-docs:</div>
<div>docs.git mlnx_ofed-4.0</div>
<div>commit 3d1b0afb7bc190ae5f362223043f76b2b45971cc</div>
<div><br>
</div>
<div>openmpi:</div>
<div>mlnx_ofed_ompi_1.8/openmpi-4.0.4rc3-1.src.rpm</div>
<div><br>
</div>
<div>opensm:</div>
<div>mlnx_ofed_opensm/opensm-5.7.0.MLNX20200721.7ccc6f6.tar.gz</div>
<div><br>
</div>
<div>openvswitch:</div>
<div>openvswitch.git mlnx_ofed_5_1</div>
<div>commit e8a86012636e058cfd48486c39afa8cbac9ed597</div>
<div>perftest:</div>
<div>mlnx_ofed_perftest/perftest-4.4-0.30.g9c50960.tar.gz</div>
<div><br>
</div>
<div>rdma-core:</div>
<div>mlnx_ofed/rdma-core.git mlnx_ofed_5_1</div>
<div>commit 77e7f704897a3bf94464d3c12ec508f1e26336fd</div>
<div><br>
</div>
<div>rshim:</div>
<div>https://github.com/Mellanox/rshim-user-space master</div>
<div>commit a70d84655d6e248141124bce1805f2c9b0426fe9</div>
<div><br>
</div>
<div>sharp:</div>
<div>mlnx_ofed_sharp/sharp-2.2.0.MLNX20200721.2fd570a.tar.gz</div>
<div><br>
</div>
<div>sockperf:</div>
<div>sockperf/sockperf-3.7-0.gita1e8e835a689.src.rpm</div>
<div><br>
</div>
<div>srp:</div>
<div>mlnx_ofed/mlnx-ofa_kernel-4.0.git mlnx_ofed_5_1</div>
<div>commit c72091bb7f69243219dda60946342385c9766aa3</div>
<div><br>
</div>
<div>ucx:</div>
<div>mlnx_ofed_ucx/ucx-1.9.0-1.src.rpm</div>
<div><br>
</div>
<br>
<br>
</div>
</div>
</body>
</html>