[ofa-general] verbs/hardware question

[Steve Wise]

Doug Ledford wrote:
> So, one of the options when creating a QP is the max inline data size.
> If I understand this correctly, for any send up to that size, the
> payload of that send will be transmitted to the receiving side along
> with the request to send.  

What it really means is the payload is DMA'd to the HW on the local side 
in the work request itself as opposed being DMA'd down in a 2nd 
transaction after the WR is DMA'd and processed.  It has no end-to-end 
significance.  Other than to reduce the latency needed to transfer the data.

> This reduces back and forth packet counts on
> the wire in the case that the receiving side is good to go, because it
> basically just responds with "OK, got it" and you're done.  

I don't think this is true.  Definitely not with iWARP.  INLINE is just 
an optimization to push small amts of data downto the local adapter as 
part of the work request, thus avoiding 2 DMA's.

> The trade
> off of course is that if there is a resource shortage on the receiving
> side, then it sends a RNR packet back, and however much payload data you
> sent over the wire with the original request to send was just wasted
> bandwidth as it was thrown away on the receiving side.
> 
> So, if my understanding of that is correct, then inline data improves
> latency and maximum bandwidth up until the point where the receiving
> side starts to have resource problems, then it wastes bandwidth and
> doesn't help latency at all.  So, if a person wanted to write their
> program to use inline data up until this point of congestion, then quit
> using it until the congestion clears, how would they go about doing
> that? 

Even though you create the QP with the inline option, only WRs that pass 
in the IBV_SEND_INLINE flag will do inline processing, so you can 
control this functionality at a per-WR basis.


> Would I have to set RNR retry count to something ridiculously
> small and take the RNR error (along with the corresponding queue flush
> and the pain that brings in terms of requeuing all the flushed events)
> and do an ibv_modify_qp to turn off inline data until some number of
> sends have completed without error?  Or is there possibly a counter
> somewhere that I can monitor?  Or should I just forget about trying to
> optimize this part of my code?
> 
> Separate question, when using an SRQ, and let's say you have more than 1
> QP associate with that SRQ, then does a single QP going into QP_ERR
> state flush the SRQ requests, or only the send requests on the QP that's
> in error?  And if you get down to only 1 QP left attached to the SRQ,
> and you then set that QP to the error state, will it flush the SRQ
> entries?  Reading everything I can on SRQs, it's not clear to me how you
> might flush one, especially since setting the SRQ itself to error state
> specifically does *not* flush the posted and unused recv requests.
> 
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> general mailing list
> general at lists.openfabrics.org
> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
> 
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general