Open Fabrics Enterprise Distribution (OFED) Version 1.3 Release Notes February 2008 =============================================================================== Table of Contents =============================================================================== 1. Overview, which includes: - OFED Distribution Rev 1.3 Contents - Supported Platforms and Operating Systems - Supported HCA and RNIC Adapter Cards and Firmware Versions - Tested Switch Platforms - Third party Test Packages - OFED sources 2. Main Changes from OFED 1.2.5 3. Fixed Bugs 4. Known Issues =============================================================================== 1. Overview =============================================================================== These are the release notes of OpenFabrics Enterprise Distribution (OFED) release 1.3 The OFED software package is composed of several software modules, and is intended for use on a computer cluster constructed as an InfiniBand subnet or iWARP network. Note: If you plan to upgrade the OFED package on your cluster, please upgrade all of its nodes to this new version. 1.1 OFED 1.3 Contents ----------------------- The OFED package contains the following components: - OpenFabrics core and ULPs: - IB HCA drivers (mthca, mlx4, ipath, ehca) - iWARP RNIC driver (cxgb3, nes) - core - Upper Layer Protocols: IPoIB, SDP, SRP Initiator and target, iSER Initiator, RDS, uDAPL and qlgc_vnic - OpenFabrics utilities: - OpenSM (OSM): InfiniBand Subnet Manager - Diagnostic tools - Performance tests - MPI: - OSU MPI stack supporting the InfiniBand and iWARP interface - Open MPI stack supporting the InfiniBand and iWARP interface - OSU MVAPICH2 stack supporting the InfiniBand and iWARP interface - MPI benchmark tests (OSU benchmarks, Intel MPI benchmarks, Presta) - Extra packages: - open-iscsi: open-iscsi initiator with iSER support - ib-bonding: Bonding driver for IPoIB interface - Sources of all software modules (under conditions mentioned in the modules' LICENSE files) - Documentation Notes: 1. SRP target is of Beta quality. 2. All other OFED components are of production quality. 3. See release notes for each package in the docs directory. 4. Any Topspin copyright belongs to Cisco Systems, Inc. 1.2 Supported Platforms and Operating Systems --------------------------------------------- o CPU architectures: - x86_64 - x86 - ppc64 - ia64 o Linux Operating Systems: - RedHat EL4 up4: 2.6.9-42.ELsmp - RedHat EL4 up5: 2.6.9-55.ELsmp - RedHat EL4 up6: 2.6.9-67.ELsmp - RedHat EL5: 2.6.18-8.el5 - RedHat EL5 up1: 2.6.18-53.el5 - Fedora C6: 2.6.18-8.fc6 * - SLES10: 2.6.16.21-0.8-smp - SLES10 SP1: 2.6.16.46-0.12-smp - SLES10 SP1 up1: 2.6.16.53-0.16-smp - OpenSuSE 10.3: 2.6.22-*-* * - kernel.org: 2.6.23 and 2.6.24 * OSes that are partially tested 1.3 HCAs and RNICs Supported ---------------------------- This release supports IB HCAs by Mellanox Technologies, Qlogic and IBM as well as iWARP RNICs by Chelsio Communications. o Mellanox Technologies HCAs (SDR and DDR Modes are Supported): - InfiniHost (fw-23108 Rev 3.5.000) - InfiniHost III Ex (MemFree: fw-25218 Rev 5.3.000 with memory: fw-25208 Rev 4.8.200) - InfiniHost III Lx (fw-25204 Rev 1.2.000) - ConnectX IB (fw-25408 Rev 2.3.000) For official firmware versions please see: http://www.mellanox.com/support/firmware_table.php o Qlogic HCAs: - QHT6040 (PathScale InfiniPath HT-460) - QHT6140 (PathScale InfiniPath HT-465) - QLE6140 (PathScale InfiniPath PE-880) o IBM HCAs: - GX Dual-port SDR 4x IB HCA - GX Dual-port SDR 12x IB HCA - GX Dual-port DDR 4x IB HCA - GX Dual-port DDR 12x IB HCA o Chelsio RNICs: - S310/S320 10GbE Storage Accelerators - R310/R320 10GbE iWARP Adapters o NetEffect RNICs: - NE020 10Gb iWARP Adapter 1.4 Switches Supported ---------------------- This release was tested with switches and gateways provided by the following companies: - Cisco - Voltaire - Qlogic - Flextronics 1.5 Third Party Packages ------------------------ The following third party packages have been tested with OFED 1.3: 1. Intel MPI, Version 3.0 - Package ID: l_mpi_p_3.0.043 2. HP-MPI 2.2.5.1 (without XRC support) 3. HP-MPI 2.2.7 (with XRC support) 1.6 OFED Sources ---------------- All sources are located under git://git.openfabrics.org/ Kernel sources: git://git.openfabrics.org/ofed_1_3/linux-2.6.git ofed_kernel User level Sources are located in all git trees as written in the BUILD_ID The kernel sources are based on Linux 2.6.24 mainline kernel. Its patches are included in the OFED sources directory. For details see HOWTO.build_ofed. =============================================================================== 2. Main Changes from OFED 1.2.5 =============================================================================== Note: For details regarding the various changes, please see the release notes for each package in the docs directory. 2.1 General changes o Kernel code based on 2.6.24 o Quality of Service support in OpenSM, CMA, IPoIB, SRP, SDP and RDS. o Added Neteffect driver (nes) o Added SRP target package: Based on Mellanox IBGD SRPT and interfaces with SCST scsi target o New verbs to support Extended RC (XRC). o Updated SA cache Integrated with sa_query module; use of SA events (GID up/down) to provide additional synchronization; User-space control for dynamic enable/disable and update 2.2 Package and install o There is a new install script. See OFED_Installation_Guide.txt for more details on the new installation and build procedures. o User space packages are now in different source RPMs (as opposed to one source RPM in previous OFED releases). o The option for a build without installing is not supported any more. o Added the script make-dist to generate tarball with kernel sources for each kernel. 2.3 IPoIB o Stateless offloads - for Datagram mode o IGMP for user-space multicast IB o NAPI is enabled default o High availability is supported via the bonding module only (ipoib tool scripts have been removed) 2.4 SDP o Keep-alive o Asynch IO o Send Zero Copy (enabled for 64K and above sends by default) 2.5 qlgc_vnic o Update for PathScale HCA 2.6 RDS o RDMA API (using FMRs) - RDS API version 3 o GA quality with Oracle 11 2.7 uDAPL o Add DAT 2.0 API run-time library and development support. uDAPL 2.0 will include IB extensions for IB rdma write with immediate data and IB atomic operations. o Both uDAPL 1.2 and 2.0 packages are provided and co-exist 2.9 iSER o open-iscsi is based on r865.12. 2.9 Management o OpenSM - Quality of Service manager - QoS implementation of the IBTA annex - QoS policy parser and support for SA (PR, MPR) - Scalability/stability improvements - Performance improvements of the routing algorithms (min-hops, up/down, lash) - Optimized port and switch tables update policy - "Native" daemon mode o Performance manager - Fetches and stores performance counters - OpenSM integrated, work in any OpenSM state (standby, disabled) - Plug-in interface o Diagnostic tools enhancements o IBSIM: IB fabric simulator (new) - Emulates user_mad layer - useful for OpenSM and management tools testing 2.10 MPI: a. OSU MVAPICH 1.0 - Asynchronous progress - Flexible message coalescing - APM support - Enhanced mpirun_rsh for scalable launching - UD-based design - Optimized and tuned collectives - Support for ConnectX - Support for QLogic/PSM b. Open MPI 1.2.5 - ConnectX support c. OSU MVAPICH2 1.0 - Message Coalescing - Hot-spot avoidance - Application-initiated systems-level check-pointing - APM support - Multi-rail support for iWARP - On-demand connection management for iWARP and uDAPL - RDMA Read - Blocking =============================================================================== 3. Fixed Bugs =============================================================================== Note: For details regarding the various fixed bugs, please see the release notes for each component in the docs directory. =============================================================================== 4. Known Issues =============================================================================== The following is a list of general limitations and known issues of the various components of the OFED 1.3 release. 1. When upgrading from an earlier OFED version, the installation script does not stop the earlier OFED version prior to uninstalling it. Workaround: Stop the old OFED stack (/etc/init.d/openibd stop) before upgrading to OFED 1.3. 2. Memory registration by the user is limited according to administrator setting. See "Pinning (Locking) User Memory Pages" in OFED_tips.txt for system configuration. 3. Fork support from kernel 2.6.12 and above is available provided that applications do not use threads. The fork() is supported as long as the parent process does not run before the child exits or calls exec(). The former can be achieved by calling wait(childpid), and the latter can be achieved by application specific means. The Posix system() call is supported. 4. The ipath driver is supported only on 64-bit platforms. 5. When installing OFED on OpenSuse or Ubuntu one should use the --without-depcheck option of the install.pl script Note: See the release notes of each component for additional issues.