[ofa-general] Setting mtu on an IB port

[Pradeep Satyanarayana]

Mike Aho wrote:
> Pradeep,
> 
> Thank you, that works well on one machine.  I was able to set the mtu on
> interface ib1 to 4096 instead of 2044.  The max_mtu from ibv_devinfo is
> still 2048 on port 2.
> On the other machine, setting the ip mtu fails if I try a number higher
> than 2044.  So which one is the correct behavior?  By the way, opensmd is
> running on the other machine.
> 
> I appreciate having the capability to do ip mtu and now have another
> command in my arsenal.
> 
> I was hoping to set the IB mtu to 4096 instead of 2048 as shown in the
> ibv_devinfo.  Any ideas on that?
> 

Mike, The issue here is likely that one of the nodes/ports may be in datagram mode
and the other one in connected mode.

cat /sys/class/net/ib0/mode

will reveal the mode for ib0. The datagram mode will limit the mtu to 2044 and
one can typically go up to 65520 with connected mode.

If you would like to switch to connected mode, you could (see caveats below) do a 

echo connected > /sys/class/net/ib0/mode

followed by the ip link command.

Caveats
-------

The above (echo) command should be okay if you are trying with a mainline kernel, ODED-1.4
or OFED-1.4.1 installation, or even Sles varaints

However, this will not work Rhel5.3 (if I remember correctly, should work for Rhel5.2).

For Rhel5.3, you need to edit the /etc/sysconfig/network-scripts/ifcfg-ib*

and insert an entry CONNECTED_MODE=yes, followed by an /etc/init.d/openibd restart.

I think that should achieve what you want. Let me know if that doesn't work.

Pradeep