[ofa-general] Setting mtu on an IB port

[Mike Aho]

Pradeep,  The configuration  was the "datagram".  Wow, that was great to 
know how to find and fix it.  Thank you very much.

So how does that value get set to "connected" or "datagram"?   The opensm 
daemon was running on the side which had datagram setting.




From:
Pradeep Satyanarayana <pradeeps at linux.vnet.ibm.com>
To:
Mike Aho/Rochester/IBM at IBMUS
Cc:
OpenFabrics General <general at lists.openfabrics.org>
Date:
06/10/2009 04:06 PM
Subject:
Re: [ofa-general] Setting mtu on an IB port



Mike Aho wrote:
> Pradeep,
> 
> Thank you, that works well on one machine.  I was able to set the mtu on
> interface ib1 to 4096 instead of 2044.  The max_mtu from ibv_devinfo is
> still 2048 on port 2.
> On the other machine, setting the ip mtu fails if I try a number higher
> than 2044.  So which one is the correct behavior?  By the way, opensmd 
is
> running on the other machine.
> 
> I appreciate having the capability to do ip mtu and now have another
> command in my arsenal.
> 
> I was hoping to set the IB mtu to 4096 instead of 2048 as shown in the
> ibv_devinfo.  Any ideas on that?
> 

Mike, The issue here is likely that one of the nodes/ports may be in 
datagram mode
and the other one in connected mode.

cat /sys/class/net/ib0/mode

will reveal the mode for ib0. The datagram mode will limit the mtu to 2044 
and
one can typically go up to 65520 with connected mode.

If you would like to switch to connected mode, you could (see caveats 
below) do a 

echo connected > /sys/class/net/ib0/mode

followed by the ip link command.

Caveats
-------

The above (echo) command should be okay if you are trying with a mainline 
kernel, ODED-1.4
or OFED-1.4.1 installation, or even Sles varaints

However, this will not work Rhel5.3 (if I remember correctly, should work 
for Rhel5.2).

For Rhel5.3, you need to edit the /etc/sysconfig/network-scripts/ifcfg-ib*

and insert an entry CONNECTED_MODE=yes, followed by an /etc/init.d/openibd 
restart.

I think that should achieve what you want. Let me know if that doesn't 
work.

Pradeep




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20090611/8be2bc2d/attachment.html>