Open Fabrics Enterprise Distribution (OFED) Version 1.2 Release Notes June 2007 =============================================================================== Table of Contents =============================================================================== 1. Overview, which includes: - OFED Distribution Rev 1.2 Contents - Supported Platforms and Operating Systems - Supported HCA and RNIC Adapter Cards and Firmware Versions - Tested Switch Platforms - Third party Test Packages - OFED sources 2. Main Changes from OFED 1.1 3. Fixed Bugs 4. Known Issues =============================================================================== 1. Overview =============================================================================== These are the release notes of OpenFabrics Enterprise Distribution (OFED) release 1.2. The OFED software package is composed of several software modules, and is intended for use on a computer cluster constructed as an InfiniBand subnet or iWARP network. Note: If you plan to upgrade the OFED package on your cluster, please upgrade all of its nodes to this new version. 1.1 OFED 1.2 Contents --------------------- The OFED package contains the following components: o OpenFabrics core and ULPs: - IB HCA drivers (mthca, ipath, ehca) - iWARP RNIC driver (cxgb3) - core - Upper Layer Protocols: IPoIB, SDP, SRP Initiator, iSER Host, RDS, uDAPL and VNIC. o OpenFabrics utilities: - OpenSM (OSM): InfiniBand Subnet Manager - Diagnostic tools - Performance tests o MPI: - OSU MPI stack supporting the InfiniBand and iWARP interface - Open MPI stack supporting the InfiniBand and iWARP interface - OSU MVAPICH2 stack supporting the InfiniBand and iWARP interface - MPI benchmark tests (OSU benchmarks, Intel MPI benchmarks, Presta) o Extra packages: - open-iscsi: open-iscsi initiator with iSER support - ib-bonding: Bonding driver for IPoIB interface o Sources of all software modules (under conditions mentioned in the modules' LICENSE files) o Documentation Notes: 1. The cxgb3 driver is in technology preview state. 2. The Virtual NIC (VNIC) driver is presented as a technology preview on OFED 1.2. 3. All other OFED components are of production quality. 4. See release notes for each package in the docs directory. 5. Any Topspin copyright belongs to Cisco Systems, Inc. 1.2 Supported Platforms and Operating Systems --------------------------------------------- o CPU architectures: - x86_64 - x86 - ia64 - ppc64 o Linux Operating Systems: - RedHat EL4 up3: 2.6.9-34.ELsmp - RedHat EL4 up4: 2.6.9-42.ELsmp - RedHat EL4 up5: 2.6.9-42.ELsmp - RedHat EL5: 2.6.9-42.ELsmp - SLES9 SP3: 2.6.5-7.244-smp - SLES10: 2.6.16.21-0.8-smp - kernel.org: 2.6.19.x and 2.6.20.x 1.3 HCAs and RNICs Supported ---------------------------- This release supports IB HCAs by Mellanox Technologies, Qlogic and IBM as well as iWARP RNICs by Chelsio Communications. o Mellanox Technologies HCAs: - InfiniHost (fw-23108 Rev 3.5.000) - InfiniHost III Ex (MemFree: fw-25218 Rev 5.2.000 with memory: fw-25208 Rev 4.8.200) - InfiniHost III Lx (fw-25204 Rev 1.2.000) The SDR and DDR modes of the InfiniHost III family are supported. For official firmware versions please see: http://www.mellanox.com/support/firmware_table.php o Qlogic HCAs: - QHT6040 (PathScale InfiniPath HT-460) - QHT6140 (PathScale InfiniPath HT-465) - QLE6140 (PathScale InfiniPath PE-880) o IBM HCAs: - GX Dual-port 4x IB HCA - GX Dual-port 12x IB HCA o Chelsio RNICs: - S310/S320 10GbE Storage Accelerators - R310E 10GbE iWARP Adapters 1.4 Switches Supported ---------------------- This release was tested with switches and gateways provided by the following companies: - Cisco - Voltaire - Qlogic - Flextronics 1.5 Third Party Packages ------------------------ The following third party packages have been tested with OFED 1.2: 1. Intel MPI, Version 3.0 - Package ID: l_mpi_p_3.0.043 2. HP MPI, Version 2.2.5 1.6 OFED Sources ---------------- Source repositories: http://www.openfabrics.org/git/ Kernel sources: ~vlad/ofed_1_2/.git User level Sources are located in all git trees starting with: ofed_1_2/ The kernel sources are based on Linux 2.6.20 mainline kernel. Its patches are included in the OFED sources directory. For details see HOWTO.build_ofed. =============================================================================== 2. Main Changes from OFED 1.1 =============================================================================== Note: For details regarding the various changes, please see the release notes for each package in the docs directory. 2.1 General changes o Kernel code based on 2.6.20 o New kernel modules: SA Cache, RDS, VNIC, bonding o High availability of SRP and IPoIB in GA level o Added iWARP support (with Chelsio driver) o MAN pages for libraries (libibverbs and librdmacm) 2.1 IPoIB o IPoIB Connected Mode o High availability support using the bonding module. 2.2 SDP o netstat is now available o Improved message BW - 10X for small messages - 5X for medium messages o Scalability - Added a memory consumption limit 2.3 SRP o High availability is now supported for all systems. 2.4 iSER o Testing more platforms (e.g., ppc64 and ia64) o Updated packages for ISCSI kernel & user components bundled with OFED. 2.5 uDAPL o Scalability features needed for Intel MPI 2.6 Libraries a. libibverbs 1.1 o Fork support (requires apps change) o Better low-level driver handling, including multiple drivers linked in statically o Documentation: man pages b. librdmacm (uCMA) 1.0 o Multicast joining from user space o UD support o Documentation: man pages 2.7 OSM o Routing improvements o Performance improvement to min hop and up/down of over an order of magnitude o New fat-tree and LASH algorithms o SA optional record support "virtually" complete o IB router enablement o SA database dump/restore 2.8 Management o Many diagnostic improvements since OFED 1.1 (see detailed RN) o ibdiagui: A GUI for ibdiagnet 2.9 Install o Default prefix directory is now /usr 2.6 MPI: a. OSU MVAPICH o Version was updated to 0.9.9 b. Open MPI o Version was updated to 1.2.1 o See http://www.open-mpi.org/svn/new.php for details c. OSU MVAPICH2 o MVAPICH2 version 0.98 was added to the OFED package. d. Common MPI setup sourcing Simple menu-driven interface to choose which MPI implementation to set as the default on a per-user and/or system-wide basis 2.7 iWARP Support o Chelsio NIC supported o Verbs and CMA APIs are the same as InfiniBand o ULPs supported - MPI (mvapich2 tested) - uDAPL o Basic Testing - uDAPL - mvapich2 - NFS-RDMA o Status: Beta =============================================================================== 3. Fixed Bugs =============================================================================== 1. OFED installation now supports installing lib32 on 64-bit systems. 2. Hotplug removal does not hang the system when the device is used by the uverbs interface. 3. MVAPICH now works on ppc64. 4. libibcm is now thread safe. Bugs fixed in each package are reported in the package's release notes. =============================================================================== 4. Known Issues =============================================================================== The following is a list of major limitations and known issues of the various components of the OFED 1.2 release. 1. Memory registration by theuser is limited according to the administrator setting. See "Pinning (Locking) User Memory Pages" in OFED_tips.txt for system configuration. 2. Fork support from kernel 2.6.12 and above is available provided that applications do not use threads. The fork() is supported as long as the parent process does not run before the child exits or calls exec(). The former can be achieved by calling wait(childpid), and the latter can be achieved by application specific means. The Posix system() call is supported. 3. The ipath driver is supported only on 64-bit platforms. 4. There are issues using Intel's MPI with the Qlogic card driver that cause failures. Note: See the release notes of each component for additional issues.