[ewg] [PATCH] pkey fix for ipoib - resubmission

[Jason Gunthorpe]

On Fri, Jun 18, 2010 at 12:56:56PM -0500, Todd Rimmer wrote:
> Jason wrote:
> > You cannot easily change the broadcast GID after starting. It gets
> > into places, and this patch does not address that. I suspect that
> > un-doing all the places the GID gets into is fairly hard..
> 
> We have successfully tested this patch by bouncing the SM with a new
> partition table config and IPoIB successfully moved to the new Pkey
> and continued working.  Within IPoIB this code was the only
> remaining place which incorrectly used the stale Pkey which the
> hardware initialized to prior to the SM programming the PKey table.

Uhm, no... the pkey is copied from the broadcast GID into many other
places. For instance it is used by the kernel to construct the
hwardware address for all IP/IPv6 joins. Without fixing up the maddr
table you break all existing multicast joins, which will completely
break IPv6, and breaks corner cases for IPv4..

Be aware that mainline and OFED are different in this regard, OFED
overrides the pkey unconditionally for multicast addresses, while
mainline does not.

Fixing the maddr table, and dealing with race issues with ongoing
joins my not be straightforward, but, IMHO, necessary for this patch
to be acceptable.

ipoib bonding had much the same problem with invalid maddrs, and a
patch was put in that flushed the maddr table in certain bond
scenarios. Perhaps something like that is a straightforward solution
here as well?

Jason