[ofa-general] Verbs questions...

[Dotan Barak]

All of the questions that you are asking can be answered by the IB spec.
But i will try to do my best to answer anyway...

Kalbag, Ashwin wrote:
> Question 1:
> Here's a section of the man page for ibv_post_send:
> "The attribute send_flags describes the properties of the WR. It is
> either 0 or the bitwise OR of one or more of the following flags:
>        IBV_SEND_FENCE  Set the fence indicator. Valid only for QPs with
> Transport Service Type IBV_QPT_RC
>        IBV_SEND_SIGNALED  Set the completion notification indicator.
> Relevant only if QP was created with sq_sig_all=0
>        IBV_SEND_SOLICITED  Set the solicited event indicator. Valid only
> for Send and RDMA Write with immediate
>        IBV_SEND_INLINE  Send data in given gather list as inline data
>               in a send WQE.  Valid only for Send and RDMA Write.  The
> L_Key will not be checked."
>
> a. What is the fence indicator?  Under what circumstances would I use
> this?
>   
Here is the line from the IB spec describing the use of the fence indicator:
 "When the Fence Indicator has been set in a Work Request, the
Send Queue shall not begin processing that Work Request until all prior
RDMA Read and Atomic Operations on that Send Queue have completed."

> b. How is a solicited event different from a signaled event?  Under what
> circumstances would I use this?
>   
Signal influence about the requestor (sender) and solicited influence 
about the responser (receiver).
There isn't any connection between them and they can used alone (only 
one of them will be set)
or together.

Signaled means that a completion will be created when did SR will be 
finished
(if the QP was created with sq_sig_all=0 this mens that only ended SR 
with the SIGNAL indicator will
 create a completion. if  sq_sig_all=1 EVERY SR will create a completion 
when it ends).

The solicited event means that a special event called solicited event 
will occur in the OTHER side
and he will be able to wait of completion event with the solicited bit set
(ibv_req_notify_cq(cq, 1) will wait until such completion is being created).
Using the solicited bit the sender can influence when the receiver side 
will handle the received completions.

> c. What is not apparent from this man page is whether the signal is
> generated on the sending side on send completion or on the receiving
> side on completion of the corresponding posted recv.  It's not
> explicitly stated, but I am assuming that the signaling refers to send
> completion on the sending side.  Is it true that regardless of whether
> the send is signaled on the sending side, it will generate a signal on
> the receiving side when the recv operation completes?
>   
I don't quite understand your question, but i think that the answered 
that i gave will answer this.
If not, can you please rephrase the question?

> Question 2:
> Say I'm trying to optimize between polling and completion event
> notification.  Could you please see whether I'm conceptualizing this
> correctly?
>
> At one extreme, you could poll continuously (without signaling sends),
> until you want to send the next message.  The polling would take care of
> recycling memory region elements on send completions, handling received
> messages for recv completions.  At the other extreme, every event would
> be signaled.  You could set sg_sig_all=1 while creating the completion
> queue, or alternatively always use the IBV_SEND_SIGNALED flag when
> posting sends.
>   
FYI: The term is every "Send Request".
And yes, completion of WRs (send or receiver request) means that
the operation was completed and you can reclaim the buffers that it 
accessed and reuse/free them.
If sg_sig_all=0 you can make sure that completions won't be created for 
all of the WR.
> To take the middle road, one possibility is to generate completions for
> every nth send.  If the messages are being sent rapidly, you can afford
> to signal fewer sends.  If sends are fewer, you would need to either
> signal more sends, say every send, or compensate by expending CPU in
> polling.  Seems like the signaling needs to be adaptive to the rate of
> sending.
>   
You are right, this is what this mechanism is all about.
> If this is the case, you still have a choice in how you process send
> completions, assuming that signals will always be generated on the
> receiving side.
In the Receiver side completions will be created for EVERY WR.

>   Even if you didn't signal every send, you could still
> process send completions by relying exclusively on signaling instead of
> polling.  If you did this, you'd need to signal at least once per "send
> queue depth", correct?  There is no urgency in processing send
> completions so long as you have some available depth in the send queue
> and elements to post sends with.  But there is urgency in processing
> recv completions.  So, it may be more optimal to process send
> completions with fewer signals, say 2 signals per send queue depth, for
> a margin of safety, so at any time, you will have half the send queue
> available.  This asymmetry would imply it is better to use separate
> completion queues for sending and receiving.  Am I right in surmising
> this?
>   
In the receiver side you have completion for every WR because you don't 
know when a message is available,
but in the sender side you knows when you send the message, so you don't 
have to create a completion
for every created SR.
You can separate the Send CQ and the Receive CQ (for example: one of 
them can be used with polling and the other
with completion events). This depend of the way you write your code. 
Maybe handling one CQ will be easier for you ...

> On the recv side, you want to process completions in a hurry and post
> more recvs, to keep up with the incoming message traffic.  Now here's
> where you might benefit from polling even after your recv event
> completion was signaled, so you avoid context switching out and back in,
> and the concomitant delay, just in case there are actually more recvs
> that completed.  Is this correct?
>   
This is Infiniband, you don't have any context switch in the data 
operations.
Post SR and Poll completions are being executed in the context you are 
in ....

You are right that incoming messages need to have more priority than 
messages that you send.

I hope that i helped a little bit.
If there is any point that i missed, you are welcome to ask again


Dotan

> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
>
> This message is intended only for the personal and confidential use of the designated recipient(s) named above.  If you are not the intended recipient of this message you are hereby notified that any review, dissemination, distribution or copying of this message is strictly prohibited.  This communication is for information purposes only and should not be regarded as an offer to sell or as a solicitation of an offer to buy any financial product, an official confirmation of any transaction, or as an official statement of Lehman Brothers.  Email transmission cannot be guaranteed to be secure or error-free.  Therefore, we do not represent that this information is complete or accurate and it should not be relied upon as such.  All information is subject to change without notice.
>
> --------
> IRS Circular 230 Disclosure:
> Please be advised that any discussion of U.S. tax matters contained within this communication (including any attachments) is not intended or written to be used and cannot be used for the purpose of (i) avoiding U.S. tax related penalties or (ii) promoting, marketing or recommending to another party any transaction or matter addressed herein.
>
>
> _______________________________________________
> general mailing list
> general at lists.openfabrics.org
> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
>
>