[ofa-general] 4GB physical memory limit on Chelsio

[pandit ib]

In iwch_provider.c, line 718, iwch_get_dma_mr() there is a comment
which seems to indicate that the Chelsio card supports registering
only 4GB of physical memory.
Is that correct?

Most of our systems have way greater memory. In fact, the HPC7000
blades we are testing on has 64GB of memory.

The problem we are running into is that our driver calls
ib_get_dma_mr() to register all physical memory.
When the driver attempts to post recv buffers it's getting -EINVAL error.

I traced it down to line 253 in iwch_qp.c.

    if (sg_list[i].addr + ((u64) sg_list[i].length) >
                    mhp->attr.va_fbo + ((u64) mhp->attr.len)) {
                        PDBG("%s %d\n", __func__, __LINE__);
                        return -EINVAL;
                }


when i printed out those values I got the following:

Jul  6 19:13:17 lv2 kernel: sg_list[i].addr faca80000,
sg_list[i].length 7360  mhp->attr.va_fbo 0 mhp->attr.len 4294967295

The length of the registration indeed seems to be 4GB in size.

Is there a way to make it work on systems with > 4GB of physical
memory. Is there a configuration parameter?

Thanks,
Ranjit