[openib-general] Mellanox HCAs: outstanding RDMAs

[Rimmer, Todd]

>  > What happens if one tries to do RDMA (say write for example) higher
> than
>  > 4 (or 128 in changed case)? does it just wait till previos
operation
>  > is completed?
>  >  I don't remember seeing any error ....it was only
>  > limited by the send Q-depth which can go much larger value.
> 
> Yes, the limit of outstanding RDMAs is not related to the send queue
> depth.  Of course you can post many more than 4 RDMAs to a send queue
> -- the HCA just won't have more than 4 requests outstanding at a time.

To further clarity, this parameter only affects the number of concurrent
outstanding RDMA Reads which the HCA will process.  Once it hits this
limit, the send Q will stall waiting for issued reads to complete prior
to initiating new reads.  It does not affect RDMA Writes.  It is very
analogous to outstanding reads parameters in PCI-X and PCIe (although
this parameters is independent from those).  The IB spec defines
ordering rules for RDMA Reads and Writes.

The number of outstanding RDMA Reads is negotiated by the CM during
connection establishment and the QP which is sending the RDMA Read must
have a value configured for this parameter which is <= the remote ends
capability.

In previous testing by Mellanox on SDR HCAs they indicated values beyond
2-4 did not improve performance (and in fact required more RDMA
resources be allocated for the corresponding QP or HCA).  Hence I
suspect a very large value like 128 would offer no improvement over
values in the 2-8 range.

Todd Rimmer