Open Fabrics Enterprise Distribution
			  Release Notes
			  
			    Rev 1.0
			   June 2006


1. Overview
===========
These are the release notes of Open Fabrics Enterprise Distribution (OFED)
release 1.0. The OFED software package is composed of several software modules
intended for use on a computer cluster constructed as an InfiniBand network.

Note: If you plan to upgrade the OFED package on your cluster, please upgrade
all its nodes to this new version.

This document is organized into the following sections:
* Overview, which includes:
	- OFED Distribution Rev 1.0 Contents.
	- Supported Platforms and Operating Systems.
	- Supported HCA Adapter Cards and Firmware Versions.
	- Tested Switch Platforms.
* OFED Utilities.
* Debug HOWTOs.
* User Memory Pages for Pinning.
* Known Issues.


1.1 OFED 1.0 Contents
---------------------
OFED package contains the following components:
  o   OpenFabrics core and ULPs:
        - HCA drivers (mthca, ipath)
        - core
        - Upper Layer Protocols: IPoIB, SDP, SRP Initiator, iSER Host, RDS and uDAPL
  o   OpenFabrics utilities:
        - OpenSM: InfiniBand Subnet Manager
        - Diagnostic tools
        - Performance tests
  o   MPI:
        - OSU MPI stack supporting the InfiniBand interface
        - Open MPI stack supporting the InfiniBand interface
        - MPI benchmark tests (OSU BW/LAT, Pallas, Presta)
  o   Sources of all software modules (under conditions mentioned in the modules'
      LICENSE files)
  o   Documentation

Notes:
1. SDP, RDS and Open MPI are in technology preview state.
2. The SRP Initiator is in beta state. All other OFED components are in production state.
3. See release notes for each package in the docs directory.
4. Any Topspin copyright belongs to Cisco Systems, Inc.



1.2 Supported Platforms and Operating Systems
---------------------------------------------
	CPU architectures:
	* x86_64
	* x86
	* ia64
	* ppc64
	
	Linux Operating Systems:
	* RedHat EL4 up2: 2.6.9-22.ELsmp
	* RedHat EL4 up3: 2.6.9-34.ELsmp
	* Fedora C4: 2.6.11-1.1369_FC4
	* SLES10 RC2: 2.6.16.16-1.6-smp (or RC 2.5 2.6.16.14-6-smp)
	* SLES10 RC1: 2.6.16.14-6-smp
	* SUSE 10 Pro: 2.6.13-15-smp
	* kernel.org: 2.6.16.x
	
1.3 HCAs Supported
------------------
This release supports HCAs from Mellanox Technologies and Qlogic.

Mellanox HCAs:
	- InfiniHost
	- InfiniHost III Ex (both modes: with memory and MemFree)
	- InfiniHost III Lx
	Both SDR and DDR mode of the InfiniHost III family are supported.

	For official FW versions please see:
	http://www.mellanox.com/support/firmware_table.php

Qlogic HCAs:
	- QHT6040 (PathScale InfiniPath HT-460)
	- QHT6140 (PathScale InfiniPath HT-465)
	- QLE6140 (PathScale InfiniPath PE-880)

1.4 Switches Supported
----------------------
This release was tested with switches and gateways provided by the following companies:
	- Cisco
	- Voltaire
	- SilverStorm
	- Flextronics


2. OFED Utilities
=================
The OFED package includes utilities under <prefix>/bin (prefix stands for the
OFED installation path).
Notes:
1. In this document we describe only a few utilities. Other utilities are useful
   too. Use the --help flag to learn about each utility.
2. The sources for all utilities are not part of the RPM installation. However,
   all sources exist in the openib-1.0.tgz tarball.

2.1 Device Information
----------------------
Device information can be obtained using several utilities:
a. ibv_devinfo
	usage:
	ibv_devinfo print the ca attributes
	Options:
	-d, --ib-dev=<dev> 	use IB device <dev> (default first device found)
	-i, --ib-port=<port> 	use port <port> of IB device (default all ports)
	-l, --list 		print only the IB devices names
	-v, --verbose 		print all the attributes of the IB device(s)
b. ibstat
	usage:
	ibstat -d(ebug) -l(ist_of_cas) -s(hort) -p(port_list)] <ca_name> [portnum]
	Examples:
	ibstat -l 		# list all IB devices
	ibstat mthca0 2 	# stat port 2 of mthca0
c. Using sysfs file system
	The driver supports the sysfs file system under: sys/class/infiniband
	Examples:
	$> ls /sys/class/infiniband/mthca0/
	board_id device fw_ver hca_type hw_rev node_desc node_guid node_type ports
	sys_image_guid
	$> cat /sys/class/infiniband/mthca0/board_id
	MT_0200000001
	$> ls /sys/class/infiniband/mthca0/ports/1/
	cap_mask counters gids lid lid_mask_count phys_state pkeys rate sm_lid sm_sl state
	$> cat /sys/class/infiniband/mthca0/ports/1/state
	4: ACTIVE

2.2 Performance Tests
---------------------
	The following performance tests are provided with the OFED release:
	1. Latency tests:
	   - ib_read_lat: RDMA read
	   - ib_write_lat: RDMA write
	   - ib_send_lat: UD, UC and RC (default) send
	2. Bandwidth tests:
	   - ib_read_bw: RDMA read
	   - ib_write_bw: RDMA write
	   - ib_send_bw: UD, UC and RC (default) send
	Usage:
	Server: <test name> <options>
	Client: <test name> <options> <server IP address>
	--help		lists the available <options>. The same options must be
			passed to both server and client.
	<server address> is an Ethernet or IPoIB address.
	
	Example: ib_send_bw
	Usage:
	ib_send_bw 			start a server and wait for connection
	ib_send_bw <host>		connect to server at <host>

	options:
	-p, --port=<port> 		listen on/connect to port <port> (default 18515)
	-d, --ib-dev=<dev> 		use IB device <dev> (default first device found)
	-i, --ib-port=<port> 		use port <port> of IB device (default 1)
	-c, --connection=<RC/UC/UD> 	connection type RC/UC/UD (default RC)
	-m, --mtu=<mtu> 		mtu size (default 1024)
	-s, --size=<size> 		size of message to exchange (default 65536)
	-a, --all 			run sizes from 2 up to 2^23
	-t, --tx-depth=<dep> 		size of tx queue (default 300)
	-n, --iters=<iters> 		number of exchanges (at least 2, default 1000)
	-b, --bidirectional 		measure bidirectional bandwidth (default is unidirectional)
	-V, --version display version number

2.3 Ping-pong Example Tests
---------------------------
	The ping-pong example tests provide basic connectivity tests. Each test
	has a help message (-h).
	- ibv_ud_pingpong
	- ibv_rc_pingpong
	- ibv_srq_pingpong
	- ibv_uc_pingpong

	Example: ibv_ud_pingpong --h
	Usage:
	ibv_ud_pingpong  		start a server and wait for connection
	ibv_ud_pingpong <host>		connect to server at <host>

	options:
	-p, --port=<port> 		listen on/connect to port <port> (default 18515)
	-d, --ib-dev=<dev> 		use IB device <dev> (default first device found)
	-i, --ib-port=<port> 		use port <port> of IB device (default 1)
	-s, --size=<size> 		size of message to exchange (default 2048)
	-r, --rx-depth=<dep> 		number of receives to post at a time (default 500)
	-n, --iters=<iters> 		number of exchanges (default 1000)
	-e, --events 			sleep on CQ events (default poll)

3. Debug HOWTOs
===============
3.1 What is Installed on the Machine
------------------------------------
3.1.1 OFED Components
The text file <prefix>/BUILD_ID provides data on all OFED components (whether 
installed or not). For example:
$> cat /usr/local/ofed/BUILD_ID
  OFED-1.0

  openib-1.0 (REV=7925)
  # User space
  https://openib.org/svn/gen2/branches/1.0/src/userspace
  # Kernel space
  https://openib.org/svn/gen2/branches/1.0/ofed/tags/1.0/linux-kernel
  Git:
  ref: refs/heads/for-2.6.17
  commit d9ec5ad24ce80b7ef69a0717363db661d13aada5

  # MPI
  mpi_osu-0.9.7-mlx2.1.0.tgz
  openmpi-1.1b1-1.src.rpm
  mpitests-1.0-0.src.rpm

3.1.2 Installed OFED Components
-------------------------------
The script /etc/infiniband/info provides data on the specific OFED installation
on this machine. For example:
$> /etc/infiniband/info
   prefix=/usr/local/ofed
   Kernel=2.6.9-22.ELsmp

   MODULES:  CONFIG_INFINIBAND=m CONFIG_INFINIBAND_USER_MAD=m
   CONFIG_INFINIBAND_USER_ACCESS=m CONFIG_INFINIBAND_ADDR_TRANS=y
   CONFIG_INFINIBAND_MTHCA=m CONFIG_IPATH_CORE=m CONFIG_INFINIBAND_IPATH=m
   CONFIG_INFINIBAND_IPOIB=m

   User level:  --kernel-version 2.6.9-22.ELsmp --kernel-sources
   /lib/modules/2.6.9-22.ELsmp/build --with-libibcm --with-libibverbs
   --with-libipathverbs --with-libmthca --with-mstflint --with-perftest


3.2 Install Driver in Debug Mode
--------------------------------
You can install the driver with debug messages enabled. To do this, enter prior
to running install.sh:
   export OPENIB_PARAMS="--debug"

Alternatively, you can place this export command in the file ofed.conf.


4. User Memory Pages for Pinning
================================
Locking memory is managed by the kernel on a per user basis. Regular users (as
opposed to root) have a limited number of pages which they may pin, where the
limit is pre-set by the administrator. Registering memory for IB verbs
requires pinning memory, thus an application cannot register more memory than
it is allowed to pin. The user can change the system setting as described below.

Note: The steps described next will allow any user in the system to lock as
much as all the memory set in the configuration files.

- Edit the file /etc/security/limits.conf and add the following two lines:
  soft memlock <number>
  hard memlock <number>
  where <number> denotes the number of KBytes that may be locked by a process.
  
  In some systems it may be possible to use "unlimited" for the size to disable
  this limit.
  
  Note: The file /etc/security/limits.conf contains further documentation.


5. Known Issues
===============
The following is a list of major limitations and known issues of the various
components of the OFED 1.0 release.

	Note: Refer to the component-specific release notes for more details.

1. Memory registration by user is limited according to the administrator
   setting. See "User Memory Pages for Pinning" for system configuration.
2. The OFED installation does not support installing lib32 on 64-bit systems
   Only libsdp is available in both lib32 and lib64.
3. On RedHat EL4 up2 and Fedora Core 4 the driver may not load properly if
   SELINUX is enforced.
   Workaround: Change the value of the parameter SELINUX in
   /etc/sysconfig/selinux from "enforcing" to "permissive" or "disabled".
4. Registration of huge page memory buffers is not supported. On some systems
   attempting to register huge pages may cause unexpected results.
5. Fork support from kernel 2.6.12 and above is available provided that
   applications do not use threads. Only system() or fork() and immediate
   exec() are supported.
6. MVAPICH and Open MPI do not work on ppc64.
7. On x86_64 systems with Fedora Core 4, if the sysfsutils-devel i386 version
   of the package is present, then a build of libibverbs will fail.

Note: See the release notes of each component for additional issues.