[ofa-general] Problems using ofed 1.4.2 and Infinipath cards

[Dave Olson]

On Wed, 26 Aug 2009, Ole Widar Saastad wrote:

| I am experiencing problems using the Infinipath cards and the OFED
| stack. (details are given below). 
| 
| It seems to be a problem somewhere when mpi packet size grows above 2k.
| This is what I recall the changeover from one transport mechanism to
| another ?

The problem is that openmpi prior to fairly recent releases had
problems with MTUs that didn't match the config file (mellanox
as well as infinipath).  Since infinipath cards are defaulted in the
config file to 4KB MTU, this showed up most on our cards, if you were
running with 2K MTU.  So you either need to fix
	mca-btl-openib-device-params.ini
on all nodes to say 2048, or override in your local configs or
command lines.

More recent versions of openmpi have this fixed (1.3.2 for sure,
maybe all 1.3, I don't remember).

Dave Olson
dave.olson at qlogic.com