Open Fabrics Enterprise Distribution (OFED)
                CHELSIO T4/T5 RNIC RELEASE NOTES
			May 2017


The iw_cxgb4 and cxgb4 modules provide RDMA and NIC support for the
Chelsio T4, T5 and T6 series adapters.  Make sure you choose the 'cxgb4'
and 'libcxgb4' options when generating your OFED rpms.

============================================
New for OFED-4.8
============================================
- Added T6 hardware support
- Added 1.16.29.0 T4-T6 firmware
- Important bug fixes

============================================
New for OFED-3.18
============================================
- Added 1.13.32.0 T4 and T5 firmware
- Important bug fixes

============================================
New for OFED-3.12
============================================
- Added T5 hardware support
- Added 1.9.23.0 T4 and T5 firmware
- Important bug fixes

===============
Supported Cards
================

T6: T62100-CR, T62100-LP-CR, T6225-CR
T5: T502-BT, T580-CR, T580-LP-CR, T520-LL-CR, T520-CR, T522-CR, T540-CR
T4: T420-CR, T440-CR, T422-CR, T404-BT, T440-LP-CR, T420-LL-CR, T420-CX

=================
Updating Firmware
=================

The OFED-4.8 driver has been tested with 1.16.29.0 and later
firmware. Version 1.16.29.0 of the firmware will automatically installed
when you load the cxgb4 driver.  The latest released firmware is always
available at service.chelsio.com inside the Unified Wire package.

If your distro/kernel cxgb4 driver supports firmware loading, you can
place the chelsio firmware image in /lib/firmware/cxgb4, then rename
it as t4fw.bin, t5fw.bin, or t6fw.bin, and unload and reload the cxgb4
module to get the new images loaded.  If this does not work, then you
can load the firmware manually as follows:

1) Move the firmware file into /lib/firmware/cxgb4/ on your system.

2) Run: ethtool -f ethX cxgb4/<fw-image-filename>

3) unload/reload cxgb4

EG:

# cp <path-to>/t6fw-1.16.29.0.bin /lib/firmware/cxgb4
# ethtool -f eth2 cxgb4/t6fw-1.16.29.0.bin
# rmmod iw_cxgb4; rmmod cxgb4; modprobe cxgb4; modprobe iw_cxgb4

NOTE: The driver package on service.chelsio.com is getting updated
periodically, including the T4/T5/T6 device firmare.  It is recommended
to use latest available firmware at service.chelsio.com.  This driver
should work with latest available firmware. In case of any issue please
contact Chelsio support at support@chelsio.com.

==============================
Setting shell for Remote Login
==============================

Users may need to set up authentication on the user account on all systems
in an MPI cluster to allow user to remotely logon or executing commands
without password.

Quick steps to set up user authentication:
- Change to user home directory
# cd
- Generate authentication key
# ssh-keygen -t rsa
- Press enter upon prompting to accept default setup
and empty password phrase.
- Create authorization file
# cd .ssh
# cat *.pub > authorized_keys
# chmod 600 authorized_keys
- Copy directory .ssh to all systems in the cluster
# cd
# scp -r .ssh remotehostname-or-ipaddress:

======================
Enabling MPA version 2
======================

The iw_cxgb4 driver in OFED-4.8 enables MPAv2 by default.  For older
versions, you should enable it manually.  Users can enable MPA version
2 by setting iw_cxgb4 module parameter as shown below

# modprobe iw_cxgb4 mpa_rev=2

MPA v2 is an enahanced RDMA connection establishment. More details are
available at:

https://tools.ietf.org/html/rfc6581

============================================
Enabling Intel and Platform MPI
============================================

For Intel MPI and Platform MPI: Enable the chelsio device by adding an
entry to /etc/dat.conf for the chelsio interface.  For instance, if your
chelsio interface name is eth2, then the following line adds a DAT version
1.2 and 2.0 devices named "chelsio" and "chelsio2" for that interface:

chelsio u1.2 nonthreadsafe default libdaplcma.so.1 dapl.1.2 "eth2 0" ""
chelsio2 u2.0 nonthreadsafe default libdaplofa.so.2 dapl.2.0 "eth2 0" ""

=============
Intel MPI:
=============

Download latest Intel MPI from the Intel website

Copy COM_L___CF8J-98P6MBWL.lic into l_mpi_p_x.y.z directory

Create machines.LINUX (list of node names) in l_mpi_p_x.y.z

Install software on every node.
#./install.sh

Register and set IntelMPI with mpi-selector (do this on all nodes).
#mpi-selector --register intelmpi --source-dir /opt/intel/impi/3.1/bin/
#mpi-selector --set intelmpi

Edit .bashrc and add these lines:

export RSH=ssh
export DAPL_MAX_INLINE=64
export I_MPI_DEVICE=rdssm:chelsio
export MPIEXEC_TIMEOUT=180
export MPI_BIT_MODE=64
ulimit -l 999999
ulimit -c unlimited
ulimit -s unlimited

Logout & log back in.

Populate mpd.hosts with node names.
Note: The hosts in this file should be Chelsio interface IP addresses. 

NOTE: I_MPI_DEVICE=rdssm:chelsio assumes you have an entry in
/etc/dat.conf named "chelsio".

NOTE: MPIEXEC_TIMEOUT value might be required to increase if heavy
traffic is going across the systems.

Contact Intel for obtaining their MPI with DAPL support.

To run Intel MPI applications:

#mpdboot -n <num nodes> -r ssh --ncpus=<num cpus>
#mpdtrace
#mpiexec -ppn <process per node> -n <num nodes> <MPI Application Path>

=============
Platform MPI
=============

Download latest Platform MPI from the IBM website

http://www.ibm.com/developerworks/downloads/im/mpi/

Install Platform MPI as:
# ./platform_mpi-09.01.02.00r.x64.bin

Choose all default settings or change accordingly.

Make sure loopback entry is present in /etc/hosts

# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

Edit .bashrc and add these lines:

export MPI_ROOT=/opt/ibm/platform_mpi/
export PATH=$MPI_ROOT/bin:/opt/bin:$PATH
export MANPATH=$MANPATH:$MPI_ROOT/share/man
ulimit -l unlimited
ulimit -s unlimited
ulimit -c unlimited

To run Platform MPI applications:

#mpirun -v -netaddr <ip-designations> -hostlist <node1>,<node2>,.....<node n> <MPI Application Path>  

=============
OpenMPI:
=============

>From OFED-3.5 package onwards OpenMPI is not a part of the OFED
package. User has to manually download and install it.

- Download latest OpenMPI released SRPM from following location.
http://www.open-mpi.org/software/ompi

- To install OpenMPI on your setup refer to the following guidelines
http://www.open-mpi.org/software/ompi/v1.6/srpm.php

OpenMPI iWARP support is only available in OpenMPI version 1.3 or greater.

Open MPI will work without any specific configuration via the openib btl.
Users wishing to performance tune the configurable options may wish to
inspect the receive queue values.  Those can be found in the "Chelsio T4"
section of mca-btl-openib-hca-params.ini.

To run OpenMPI applications:

 mpirun --host <node1>,<node2> -mca btl openib,sm,self <OpenMPI Application Path>

=============
MVAPICH2:
=============

>From OFED-3.5 package onwards MVAPICH2 is not a part of the OFED
package. User has to manually download and install it.

- Download latest MVAPICH2 released SRPM from following location.
http://mvapich.cse.ohio-state.edu/download/mvapich2/

- To install MVAPICH2 on your setup do the following
./configure --prefix=/usr --sysconfdir=/etc --libdir=/usr/lib64 && make &&
make install

The following env vars enable MVAPICH2 version 1.4-2.  Place these
in your user env after installing and setting up MVAPICH2 MPI:

export MVAPICH2_HOME=/usr/mpi/gcc/mvapich2-1.4/
export MV2_USE_IWARP_MODE=1
export MV2_USE_RDMA_CM=1

On each node, add this to the end of /etc/profile.

 ulimit -l 999999

On each node, add this to the end of /etc/init.d/sshd and restart sshd.

 ulimit -l 999999
 % service sshd restart

Verify the ulimit changes worked. These should show '999999':

 % ulimit -l
 % ssh <peer> ulimit -l

Note: You may have to restart sshd a few times to get it to work.

Create mpd.hosts with list of hostname or ipaddrs in the cluster. They
should be names/addresses that you can ssh to without passwords. (See
Passwordless SSH Setup).

On each node, create /etc/mv2.conf with a single line containing the
IP address of the local T4 interface. This is how MVAPICH2 picks which
interface to use for RDMA traffic.

On each node, edit /etc/hosts file. Comment the entry if there is an
entry with 127.0.0.1 IP Address and local host name. Add an entry for
corporate IP address and local host name (name that you have given in
mpd.hosts file) in /etc/hosts file.

To run MVAPICH2 application:

 mpirun_rsh -ssh -np 8 -hostfile mpd.hosts <MVAPICH2 Application Path>

============================================
Testing connectivity with ping and rping:
============================================

Configure the ethernet interfaces for your T4 device. After you modprobe
iw_cxgb4 you will see ethernet interfaces for the T4 device.  Configure
them with an appropriate ip address, netmask, etc.  You can use the
Linux ping command to test basic connectivity via the T4 interface.

To test RDMA, use the rping command that is included in the
librdmacm-utils rpm:

On the server machine:

# rping -s -p 9999

On the client machine:

# rping -c -VvC10 -a server_ip_addr -p 9999

You should see ping data like this on the client:

ping data: rdma-ping-0: ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqr
ping data: rdma-ping-1: BCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrs
ping data: rdma-ping-2: CDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrst
ping data: rdma-ping-3: DEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstu
ping data: rdma-ping-4: EFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuv
ping data: rdma-ping-5: FGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvw
ping data: rdma-ping-6: GHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwx
ping data: rdma-ping-7: HIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxy
ping data: rdma-ping-8: IJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz
ping data: rdma-ping-9: JKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyzA
client DISCONNECT EVENT...