[ofa-general] Re: 0-length RDMA Read

[Philip Frey1]

Steve,

many thanks for your quick response!

I read about this P2P feature but did not know how to activate it. In the 
meantime I decided
to do a 0-length RDMA Write from the initiator - that worked right away 
and has the same effect.

Cheers,
 Philip

Steve Wise <swise at opengridcomputing.com> wrote on 01/28/2009 04:33:05 PM:

> [image removed] 
> 
> Re: 0-length RDMA Read
> 
> Steve Wise 
> 
> to:
> 
> Philip Frey1
> 
> 01/28/2009 04:34 PM
> 
> Cc:
> 
> general, Felix Marti
> 
> This looks like a bug.  The lib assumes an SGE entry will be provided.
> 
> A workaround for now is to set num_sge to 1 and initialize the sge entry 
to:
> 
> sge.addr 0
> sge.lkey 2
> sge.length 0
> rkey 2
> remote_addr 0
> 
> I'll fix this in libcxgb3 to allow num_sge == 0 to mean 0B read.  Also, 
> right now you need to specify non-zero (and yet still valid possible) 
> lkey and rkey values.  I'll fix this too so if length is 0 or num_sge is 

> 0, then the library will create a valid 0B read request for you ignoring 

> the other fields.  I opened bug 1496 for this: 
> https://bugs.openfabrics.org/show_bug.cgi?id=1496
> 
> Lemme know if you want a new libcxgb3 tarball with the fix.
> 
> By the way, as of ofed-1.3.1 and 2.6.27 kernels, iw_cxgb3 supports a 
> mode where it handles this client-must-send-first issue for you.  There 
> is a module option called peer2peer.  Set it to 1 and all subsequent 
> connections will handle this by doing a 0B read from the client.
> 
> 'echo 1 > /sys/module/iw_cxgb3/parameters/peer2peer'  will do the 
trick...
> 
> Steve.
> 
> 
> Philip Frey1 wrote:
> >
> > Hi,
> >
> > since for iWARP, that after the MPA connection establishment, the
> > MPA initiator must send the first FPDU, I wanted to do that using a 
> > 0-length
> > RDMA Read. When using the T3 Chelsio RNIC, I end up with a 
segmentation
> > fault from the libcxgb3 (function t3b_post_send).
> >
> > I was trying to post a 0-lenght WR for RDMA Read like this:
> >
> > struct ibv_send_wr        wr;
> >
> > wr.wr_id = 1;
> > wr.next = NULL;
> > wr.sg_list = NULL;
> > wr.num_sge = 0;
> > wr.wr.rdma.remote_addr = 0;
> > wr.wr.rdma.rkey = 0;
> > wr.opcode = IBV_WR_RDMA_READ;
> > wr.send_flags = IBV_SEND_SIGNALED;
> >
> > ibv_post_send(qp, &wr, &bad_wr);
> >
> > Question1: Are 0-length RDMA Reads supported at all by the T3?
> > Question2: If they are, how do I have to write a correct send WR?
> >
> > Many thanks for your advice,
> >  Philip
> >
> >
> >
> > -- 
> >   Philip Frey 
> >   IBM Zurich Research Laboratory
> >   Saumerstrasse 4                                   |  Phone: +41 44 
> > 724 8613
> >   CH-8803 Rueschlikon/Switzerland  |  Email: phf at zurich.ibm.com 
> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20090129/618cac87/attachment.html>