[ofa-general] Re: [PATCH 0/3] ib/ipoib: Enable IPoIB-UD 4K MTU support

[Shirley Ma]

On Thu, 2008-01-31 at 09:32 +0200, Or Gerlitz wrote:
> Hi Shirley,
> 
> Just to make sure, can you confirm that this patch set is not
> dependent 
> on the below patch which is part of ofed but was never submitted to
> the 
> upstream ipoib driver for inclusion?

	No, this patchset is not dependent on any OFED patches. It's a pure
patch set for 2.6.25 kernel. I have another version of this patchset
which is built against OFED-1.3-RC2. I will update it to OFED-1.3-RC3. I
hope I can get a quick ack for this patchset from maintainers to agree
with this approach. There are around 1.5-2 times better performance I
can see to use 4K MTU for IPoIB-UD. I will resumit this patchset
tomorrow. You should wait for the new patchset since I have found some
minor problem when I splitted these patches.

> Also, can you share with what SM have you checked this, did you had
> to 
> patch or run it with non-default param, more, what was the 
> configuration, specifically what switch was used and any
> instrumentation 
> you have made to the switch FW, thanks. 

	One of the reason this patchset was not be able to submit earlier was
because of the SW support. I couldn't do a full test without SW supports
4K MTU. The SW firmware needs to be update to allow IPoIB broadcast
group to be able to create 4096 MTU size. There are two requirements to
the switch from SW perspective:
1. SW ports are able to configure to 4096 MTU size.
2. SW default IPoIB broadcast group is able to configure to 4096 MTU
size. The default IPoIB broadcast group MTU can't exceed SW ports MTU
size. 

The way to enable IPoIB 4K MTU is:
1. set SW ports to 4K MTU
2. set SM default IPoIB broadcast group MTU size as 4K. 

	You could disable or enable IPoIB broadcast group when starting SM. If
you don't enable IPoIB default broadcast group when starting SM, the
first node in the subnet will come up and create a broadcast group with
2K MTU for this subnet. It makes sense since the node doesn't know the
whole subnet link MTU size. So it's better to create a default 2K MTU.
If you enable IPoIB default broadcast group when starting SM, if the MTU
size is 2K, then all nodes in the cluster can join the subnet and the
IPoIB subnet link MTU size will be set to 2K. If the broadcast group MTU
size is 4K, then only nodes with 4K MTU can join this IPoIB subnet.

I am not sure that's what you are looking for. Let me know if anything
is unclear.

thanks
Shirley