[ofw] RE: bugcheck in mlx4_bus

[Sean Hefty]

After enabling driver verifier on 2008 HPC, I get the a page fault in
mlx4_bus:ib_get_cached_pkey().  The stack trace is:

mlx4_bus!ib_get_cached_pkey
mlx4_have!mlnx_query_ca
winmad!WmQueryCaAttributes
winmad!WmAddCa
winmad!WvPowerD0Entry

The faulting code is marked below >>>

int ib_get_cached_pkey(struct ib_device *device,
		       u8                port_num,
		       int               index,
		       __be16           *pkey)
{
	struct ib_pkey_cache *cache;
	unsigned long flags;
	int ret = 0;

	if (mlx4_is_barred(device->dma_device))
		return -EFAULT;

	if (port_num < start_port(device) || port_num > end_port(device))
		return -EINVAL;

	read_lock_irqsave(&device->cache.lock, &flags);

	cache = device->cache.pkey_cache[port_num - start_port(device)];

	if (index < 0 || index >= cache->table_len)
		ret = -EINVAL;
	else
>>>		*pkey = cache->table[index];

	read_unlock_irqrestore(&device->cache.lock, flags);

	return ret;
}

read_lock_irqsave maps to spinlock_irqsave which maps to spin_lock which maps to
cl_spinlock_acquire which maps to KeAcquireSpinLock() which is, gasp!, an actual
call.

The *pkey is allocated as part of a buffer by winmad!WmQueryCaAttributes as
PagedPool, which leads to the page fault.  Winverbs uses the interface in the
same way. 

Obviously, winmad and winverbs can change to allocate the attributes from
NonPagedPool, but that doesn't seem like the right fix.  query_ca() is a
blocking call, so one would expect that providing a buffer from paged pool
should work.  All of the ib_get_cached_* calls have this issue, but for
query_ca() it looks like get_cached_pkey() and get_cached_gid() are the only
ones of interest.

I think the easiest fix for this is to have ib_get_cached_* calls assign their
values outside of any locking.  Something like:

lock();
pkey = cache->table[index];
unlock();
*pkey = pkey;

If there's no objection, I'll create a patch along these lines.

- Sean