[ofa-general] 4GB physical memory limit on Chelsio

[Steve Wise]

pandit ib wrote:
> In iwch_provider.c, line 718, iwch_get_dma_mr() there is a comment
> which seems to indicate that the Chelsio card supports registering
> only 4GB of physical memory.
> Is that correct?
>
> Most of our systems have way greater memory. In fact, the HPC7000
> blades we are testing on has 64GB of memory.
>
> The problem we are running into is that our driver calls
> ib_get_dma_mr() to register all physical memory.
> When the driver attempts to post recv buffers it's getting -EINVAL error.
>
>   

> I traced it down to line 253 in iwch_qp.c.
>
>     if (sg_list[i].addr + ((u64) sg_list[i].length) >
>                     mhp->attr.va_fbo + ((u64) mhp->attr.len)) {
>                         PDBG("%s %d\n", __func__, __LINE__);
>                         return -EINVAL;
>                 }
>
>
> when i printed out those values I got the following:
>
> Jul  6 19:13:17 lv2 kernel: sg_list[i].addr faca80000,
> sg_list[i].length 7360  mhp->attr.va_fbo 0 mhp->attr.len 4294967295
>
> The length of the registration indeed seems to be 4GB in size.
>
> Is there a way to make it work on systems with > 4GB of physical
> memory. Is there a configuration parameter?
>
> Thanks,
> Ranjit
>   
Hey Ranjit,

Sorry, this is limitation in the current Chelsio HW (T3).

You cannot use get_dma_mr to register memory above 4GB.  It just doesn't 
work for T3.  In fact, get_dma_mr just maps dma addresses 0..4GB-1.  So 
if your system is using a iommu and mapping bus addresses above 4GB, it 
won't work at all.  Chelsio's next part will overcome this limitation.  
In the meantime, you have to use reg_phys_mr or the fast_register work 
request to register specific regions on demand.  You can see how this is 
done by investigating the NFS RDMA transport code.

Steve.

> _______________________________________________
> general mailing list
> general at lists.openfabrics.org
> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
>