[ofa-general] ib_umad method mask problems on big-endian 64-bit archs

[Sasha Khapyorsky]

On 14:32 Mon 20 Aug     , Roland Dreier wrote:
> 
> I'd like to merge this, but unfortunately there is the complication that
> opensm/libvendor/osm_vendor_ibumad.c has this:
> 
> 	static int set_bit(int nr, void *method_mask)
> 	{
> 		int mask, retval;
> 		long *addr = method_mask;
> 	
> 		addr += nr >> 5;
> 		mask = 1 << (nr & 0x1f);
> 		retval = (mask & *addr) != 0;
> 		*addr |= mask;
> 		return retval;
> 	}
> 
> which is actually wrong in general (if sizeof long is 8, then
> offsetting by nr>>5 ends up making the subscript twice what it should
> be), but all defined IB MAD methods are less than 32 so it ends up
> being OK, and actually works for 64-bit big-endian userspace on a
> 64-bit kernel.  So the patch below would mean that 64-bit osm binaries
> on ppc64 would break (32-bit osm binaries on powerpc are of course
> broken with a 64-bit kernel).  Still, perhaps the right thing to do is
> to just fix the opensm code at the same time?
> 
> Opinions?

It looks that using uint32_t for addr in set_bit() function is sufficient
fix. But for ppc64 this means that new OpenSM will break with old
kernels, probably we will need to put some ugly #ifdef in
osm_vendor_ibumad.c...

Sasha