[ofa-general] Re: 0-length RDMA Read
Steve Wise
swise at opengridcomputing.com
Wed Jan 28 07:33:05 PST 2009
This looks like a bug. The lib assumes an SGE entry will be provided.
A workaround for now is to set num_sge to 1 and initialize the sge entry to:
sge.addr 0
sge.lkey 2
sge.length 0
rkey 2
remote_addr 0
I'll fix this in libcxgb3 to allow num_sge == 0 to mean 0B read. Also,
right now you need to specify non-zero (and yet still valid possible)
lkey and rkey values. I'll fix this too so if length is 0 or num_sge is
0, then the library will create a valid 0B read request for you ignoring
the other fields. I opened bug 1496 for this:
https://bugs.openfabrics.org/show_bug.cgi?id=1496
Lemme know if you want a new libcxgb3 tarball with the fix.
By the way, as of ofed-1.3.1 and 2.6.27 kernels, iw_cxgb3 supports a
mode where it handles this client-must-send-first issue for you. There
is a module option called peer2peer. Set it to 1 and all subsequent
connections will handle this by doing a 0B read from the client.
'echo 1 > /sys/module/iw_cxgb3/parameters/peer2peer' will do the trick...
Steve.
Philip Frey1 wrote:
>
> Hi,
>
> since for iWARP, that after the MPA connection establishment, the
> MPA initiator must send the first FPDU, I wanted to do that using a
> 0-length
> RDMA Read. When using the T3 Chelsio RNIC, I end up with a segmentation
> fault from the libcxgb3 (function t3b_post_send).
>
> I was trying to post a 0-lenght WR for RDMA Read like this:
>
> struct ibv_send_wr wr;
>
> wr.wr_id = 1;
> wr.next = NULL;
> wr.sg_list = NULL;
> wr.num_sge = 0;
> wr.wr.rdma.remote_addr = 0;
> wr.wr.rdma.rkey = 0;
> wr.opcode = IBV_WR_RDMA_READ;
> wr.send_flags = IBV_SEND_SIGNALED;
>
> ibv_post_send(qp, &wr, &bad_wr);
>
> Question1: Are 0-length RDMA Reads supported at all by the T3?
> Question2: If they are, how do I have to write a correct send WR?
>
> Many thanks for your advice,
> Philip
>
>
>
> --
> Philip Frey
> IBM Zurich Research Laboratory
> Saumerstrasse 4 | Phone: +41 44
> 724 8613
> CH-8803 Rueschlikon/Switzerland | Email: phf at zurich.ibm.com
More information about the general
mailing list