[ofa-general] SDP and iWARP

[Craig Prescott]

Steve Wise wrote:
> Craig Prescott wrote:
>>
>> Hi Steve;
>>
>> The SDP socket gets an associated mr when sdp_init_qp() calls 
>> ib_get_dma_mr().  It looks to me like this drills down into
>> the provider layer, which will ultimately end up calling
>> build_phys_page_list() from iwch_register_phys_mem().
>>
>> Unfortunately, when I try to look at the ib_mr_attrs via
>> ib_query_mr(), the call fails.
>>
>> When sdp_post_recv() calls ib_post_recv(), it looks to me
>> like a DMA mapping has been set up between the SDP private
>> receive buffers and card.  The receive buffers are kmalloc'd
>> in sdp_init_qp().
>>
>> I hope I have this right.  But it sounds like it is possible
>> I am hitting both issues you describe.
>>
>> I guess one way to check is to drop my test nodes down to 4GB
>> or less, right?  They currently have 16GB.
>>
> 
> Drop them down to 1 or 2GB and try it.  4GB still requires the iommu to 
> remap things above 4GB.

Awesome ;-)  rdma_accept() now returns zero and no more complaints
about opcodes and such, and the server now gets to 
RDMA_CM_EVENT_ESTABLISHED.  Thanks!

Of course, the client panic'd at this point.  Little by little...

> Sorry about that.  I forgot about the 4GB limitation and get_dma_mr(). I 
> guess the chelsio driver should really just fail the get_dma_mr() call 
> since it doesn't properly support it.
> 
> There is one other experiment you could try.  You could try using lkey 0 
> for any sgl used in a send or receive work request.  This maps to the 
> zero stag in iwarp lingo.  But I haven't tested that yet :)

I'll try to do this tomorrow.

Thanks again!
Craig