[ofa-general] Write combining support in OFED 1.5

[Jack Morgenstein]

Hi Roland,

I've been looking at the write-combining support in the 2.6.30 kernel, and it looks good.
There is also a good solution for PPC write combining support in the kernel
(adding #define pgprot_writecombine  pgprot_noncached_wc to file arch/powerpc/include/asm/pgtable.h,
 per e-mail correspondence with kernel ppc people in ibm).

However, I am still missing some means of determining whether or not write_combining is supported
on a given host.

From the write-combining support in OFED 1.4:

static struct ib_ucontext *mlx4_ib_alloc_ucontext(struct ib_device *ibdev,
						  struct ib_udata *udata)
{
	struct mlx4_ib_dev *dev = to_mdev(ibdev);
	struct mlx4_ib_ucontext *context;
	struct mlx4_ib_alloc_ucontext_resp resp;
	int err;

	resp.qp_tab_size      = dev->dev->caps.num_qps;

===>	if (mlx4_wc_enabled()) {
		resp.bf_reg_size      = dev->dev->caps.bf_reg_size;
		resp.bf_regs_per_page = dev->dev->caps.bf_regs_per_page;
	} else {
		resp.bf_reg_size      = 0;
		resp.bf_regs_per_page = 0;
	}

The issue is that if a given platform does not support write-combining, we do not wish to 
enable use of blueflame in userspace for ConnectX.  I have not yet come up with any good,
consistent, way of determining if write-combining is supported.

I thought of possibly comparing the result returned by pgprot_writecombine(prot) to that
returned by pgprot_noncached(prot) -- if they are identical, then assume that write-combining
is not supported.
(pgprot_noncached() is the default mapping of pgprot_writecombine if it is not defined under
 the arch directory -- see file include/linux/pgtable.h).

This has a problem, however, in that I have no way of determining what value of "prot" to
use when doing this comparison -- there may be some architectures which use bits of the
prot structure to determine per specific call whether or not to use write-combining
(i.e., pgprot_writecombine(prot) could invoke pgprot_noncached(prot) if certain bits were
set in the prot structure, or return a write-combining prot value if those bits are not set).

Using a zeroed-out pgprot structure in the comparison, for example, may not be appropriate.
(we may be allowing blueflame when it should not be, or preventing blueflame when it should be allowed).

Do you have any ideas for how to determine if in fact write-combining is available?

-Jack