[ofa-general] Verbs questions...

[Kalbag, Ashwin]

Question 1:
Here's a section of the man page for ibv_post_send:
"The attribute send_flags describes the properties of the WR. It is
either 0 or the bitwise OR of one or more of the following flags:
       IBV_SEND_FENCE  Set the fence indicator. Valid only for QPs with
Transport Service Type IBV_QPT_RC
       IBV_SEND_SIGNALED  Set the completion notification indicator.
Relevant only if QP was created with sq_sig_all=0
       IBV_SEND_SOLICITED  Set the solicited event indicator. Valid only
for Send and RDMA Write with immediate
       IBV_SEND_INLINE  Send data in given gather list as inline data
              in a send WQE.  Valid only for Send and RDMA Write.  The
L_Key will not be checked."

a. What is the fence indicator?  Under what circumstances would I use
this?
b. How is a solicited event different from a signaled event?  Under what
circumstances would I use this?
c. What is not apparent from this man page is whether the signal is
generated on the sending side on send completion or on the receiving
side on completion of the corresponding posted recv.  It's not
explicitly stated, but I am assuming that the signaling refers to send
completion on the sending side.  Is it true that regardless of whether
the send is signaled on the sending side, it will generate a signal on
the receiving side when the recv operation completes?

Question 2:
Say I'm trying to optimize between polling and completion event
notification.  Could you please see whether I'm conceptualizing this
correctly?

At one extreme, you could poll continuously (without signaling sends),
until you want to send the next message.  The polling would take care of
recycling memory region elements on send completions, handling received
messages for recv completions.  At the other extreme, every event would
be signaled.  You could set sg_sig_all=1 while creating the completion
queue, or alternatively always use the IBV_SEND_SIGNALED flag when
posting sends.

To take the middle road, one possibility is to generate completions for
every nth send.  If the messages are being sent rapidly, you can afford
to signal fewer sends.  If sends are fewer, you would need to either
signal more sends, say every send, or compensate by expending CPU in
polling.  Seems like the signaling needs to be adaptive to the rate of
sending.

If this is the case, you still have a choice in how you process send
completions, assuming that signals will always be generated on the
receiving side.  Even if you didn't signal every send, you could still
process send completions by relying exclusively on signaling instead of
polling.  If you did this, you'd need to signal at least once per "send
queue depth", correct?  There is no urgency in processing send
completions so long as you have some available depth in the send queue
and elements to post sends with.  But there is urgency in processing
recv completions.  So, it may be more optimal to process send
completions with fewer signals, say 2 signals per send queue depth, for
a margin of safety, so at any time, you will have half the send queue
available.  This asymmetry would imply it is better to use separate
completion queues for sending and receiving.  Am I right in surmising
this?

On the recv side, you want to process completions in a hurry and post
more recvs, to keep up with the incoming message traffic.  Now here's
where you might benefit from polling even after your recv event
completion was signaled, so you avoid context switching out and back in,
and the concomitant delay, just in case there are actually more recvs
that completed.  Is this correct?

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

This message is intended only for the personal and confidential use of the designated recipient(s) named above.  If you are not the intended recipient of this message you are hereby notified that any review, dissemination, distribution or copying of this message is strictly prohibited.  This communication is for information purposes only and should not be regarded as an offer to sell or as a solicitation of an offer to buy any financial product, an official confirmation of any transaction, or as an official statement of Lehman Brothers.  Email transmission cannot be guaranteed to be secure or error-free.  Therefore, we do not represent that this information is complete or accurate and it should not be relied upon as such.  All information is subject to change without notice.

--------
IRS Circular 230 Disclosure:
Please be advised that any discussion of U.S. tax matters contained within this communication (including any attachments) is not intended or written to be used and cannot be used for the purpose of (i) avoiding U.S. tax related penalties or (ii) promoting, marketing or recommending to another party any transaction or matter addressed herein.