[openib-general] Re: Mellanox HCAs: outstanding RDMAs

[Talpey, Thomas]

At 03:43 AM 6/6/2006, Michael S. Tsirkin wrote:
>Quoting r. Talpey, Thomas <Thomas.Talpey at netapp.com>:
>> Semantically, the provider is not required to provide any such flow control
>> behavior by the way. The Mellanox one apparently does, but it is not
>> a requirement of the verbs, it's a requirement on the upper layer. If more
>> RDMA Reads are posted than the remote peer supports, the connection
>> may break.
>
>This does not sound right. Isn't this the meaning of this field:
>"Initiator Depth: Number of RDMA Reads & atomic operations
>outstanding at any time"? Shouldn't any provider enforce this limit?

The core spec does not require it. An implementation *may* enforce it,
but is not *required* to do so. And as pointed out in the other message,
there are repercussions of doing so.

I believe the silent queue stalling is a bit of a time bomb for upper layers,
whose implementers are quite likely unaware of the danger. I greatly
prefer an implementation which simply sends the RDMA Read request,
resulting in a failed (but unblocked!) connection. Silence is a very
dangerous thing, no matter how helpful the intent.

Tom.