[ofa-general] verbs/hardware question

[Doug Ledford]

So, one of the options when creating a QP is the max inline data size.
If I understand this correctly, for any send up to that size, the
payload of that send will be transmitted to the receiving side along
with the request to send.  This reduces back and forth packet counts on
the wire in the case that the receiving side is good to go, because it
basically just responds with "OK, got it" and you're done.  The trade
off of course is that if there is a resource shortage on the receiving
side, then it sends a RNR packet back, and however much payload data you
sent over the wire with the original request to send was just wasted
bandwidth as it was thrown away on the receiving side.

So, if my understanding of that is correct, then inline data improves
latency and maximum bandwidth up until the point where the receiving
side starts to have resource problems, then it wastes bandwidth and
doesn't help latency at all.  So, if a person wanted to write their
program to use inline data up until this point of congestion, then quit
using it until the congestion clears, how would they go about doing
that?  Would I have to set RNR retry count to something ridiculously
small and take the RNR error (along with the corresponding queue flush
and the pain that brings in terms of requeuing all the flushed events)
and do an ibv_modify_qp to turn off inline data until some number of
sends have completed without error?  Or is there possibly a counter
somewhere that I can monitor?  Or should I just forget about trying to
optimize this part of my code?

Separate question, when using an SRQ, and let's say you have more than 1
QP associate with that SRQ, then does a single QP going into QP_ERR
state flush the SRQ requests, or only the send requests on the QP that's
in error?  And if you get down to only 1 QP left attached to the SRQ,
and you then set that QP to the error state, will it flush the SRQ
entries?  Reading everything I can on SRQs, it's not clear to me how you
might flush one, especially since setting the SRQ itself to error state
specifically does *not* flush the posted and unused recv requests.

-- 
Doug Ledford <dledford at redhat.com>
              GPG KeyID: CFBFF194
              http://people.redhat.com/dledford

Infiniband specific RPMs available at
              http://people.redhat.com/dledford/Infiniband
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20071011/d90ec8f6/attachment.sig>