[openib-general] Re: Mellanox HCAs: outstanding RDMAs

[Michael S. Tsirkin]

Quoting r. Manpreet Singh <manpreet at gmail.com>:
> Subject: Re: Mellanox HCAs: outstanding RDMAs
> 
> We have seen this happen over an IB analyzer. Recompiling the mthca driver with a high value like 64 or 128 works around this problem.
> When the condition hits, the HCA receiving the 4+ RDMAs generates an invalid request error.

Posting more read work requests than might be outstanding simultaneously on the
wire is not an error.

I think the fact you are getting an error means you are configuring
max_rd_atomic/max_dest_rd_atomic on the local versus remote side incorrectly
(these represent the Number of responder resources for RDMA Read/atomic ops and
Number of Outstanding RDMA Read/atomic ops at destination, respectively).

If so this is a bug in ULP, working around it by increasing the number
of credits on both sides does not seem like the right thing to do.

See 12.7.29 RESPONDER RESOURCES, and 12.7.30 INITIATOR DEPTH.

-- 
MST