[ofa-general] SDP and iWARP

[Steve Wise]

Craig Prescott wrote:
> 
> Hi Steve;
> 
> The SDP socket gets an associated mr when sdp_init_qp() calls 
> ib_get_dma_mr().  It looks to me like this drills down into
> the provider layer, which will ultimately end up calling
> build_phys_page_list() from iwch_register_phys_mem().
> 
> Unfortunately, when I try to look at the ib_mr_attrs via
> ib_query_mr(), the call fails.
> 
> When sdp_post_recv() calls ib_post_recv(), it looks to me
> like a DMA mapping has been set up between the SDP private
> receive buffers and card.  The receive buffers are kmalloc'd
> in sdp_init_qp().
> 
> I hope I have this right.  But it sounds like it is possible
> I am hitting both issues you describe.
> 
> I guess one way to check is to drop my test nodes down to 4GB
> or less, right?  They currently have 16GB.
> 

Drop them down to 1 or 2GB and try it.  4GB still requires the iommu to 
remap things above 4GB.

Sorry about that.  I forgot about the 4GB limitation and get_dma_mr(). 
I guess the chelsio driver should really just fail the get_dma_mr() call 
since it doesn't properly support it.

There is one other experiment you could try.  You could try using lkey 0 
for any sgl used in a send or receive work request.  This maps to the 
zero stag in iwarp lingo.  But I haven't tested that yet :)

Steve.