[openib-general] Re: Gen2 User verbs usage

[Roland Dreier]

>>>>> "Sayantan" == Sayantan Sur <surs at cse.ohio-state.edu> writes:

    Sayantan> Hi Roland, I have some questions regarding the usage of
    Sayantan> the new Gen2 verbs.

    Sayantan> 1. Polling CQ : I notice that this verb is little
    Sayantan> different from VAPI_poll_cq, in the sense that it
    Sayantan> accepts a parameter to poll for multiple completion
    Sayantan> entries. So, if I have a statement like:

    Sayantan>     497 ne = ibv_poll_cq(hca.cq, 1, &wc);

    Sayantan> I want to poll for one completion. Does `ne' hold the
    Sayantan> return status or number of elements actually pulled out
    Sayantan> of CQ?

Sorry, this should be documented better in the userspace library.  The
semantics are identical to the ib_poll_cq() function in the kernel:

 * Poll a CQ for (possibly multiple) completions.  If the return value
 * is < 0, an error occurred.  If the return value is >= 0, it is the
 * number of completions returned.  If the return value is
 * non-negative and < num_entries, then the CQ was emptied.

    Sayantan> 2. Posting RDMA write : Do these statements for
    Sayantan> preparing a RDMA write IB descriptor make sense?

    Sayantan>     472     sr_desc.send_flags = IBV_SEND_SIGNALED;
    Sayantan>     473     sr_desc.opcode = IBV_WR_RDMA_WRITE;
    Sayantan>     474     sr_desc.wr_id = 0;
    Sayantan>     475     sr_desc.num_sge = 1;

    Sayantan>     477     sr_desc.sg_list = &(sg_entry);

    Sayantan>     479     sr_desc.wr.rdma.remote_addr = (uintptr_t) (rbuf.buf);
    Sayantan>     480     sr_desc.wr.rdma.rkey = rbuf.rkey;

    Sayantan>     483     sg_entry.addr = (uintptr_t) (lbuf.buf);
    Sayantan>     484     sg_entry.length = len;
    Sayantan>     485     sg_entry.lkey = lbuf.mr->lkey;

    Sayantan> Essentially, I don't understand what the `send_flags'
    Sayantan> field means.

Yes, this all makes sense.  The send_flags field can hold any
combination (|'ed together) of the flags IBV_SEND_FENCE,
IBV_SEND_SIGNALED, IBV_SEND_SOLICITED and IBV_SEND_INLINE.  FENCE
means that strict ordering will be enforced, as described in section
10.8 of the IB spec.  IBV_SEND_SIGNALED means that a CQ entry will be
generated when the send is completed (this flag is ignored if the QP
is created with sq_sig_all != 0, since all sends will generate CQ
entries anyway).  SOLICITED means that the solicited bit will be set
in the message so that the remote side will receive a solicited
completion event.  INLINE means the verbs driver should try to copy
the data directly into the send work request to reduce latency.

    Sayantan> On a side note, if you think that this sort of
    Sayantan> discussion is useful in openib-general, please feel free
    Sayantan> to Cc to that list.

Yes, I definitely think these questions should go through the mailing
list so that all the subscribers (plus any future archive searchers!)
can learn from the answers.

Thanks,
  Roland