<html>
<body>
<font size=3><br>
Whether iWARP or IB, there is a fixed number of RDMA Requests allowed to
be outstanding at any given time. If one posts more RDMA Read
requests than the fixed number, the transmit queue is stalled. This
is documented in both technology specifications. It is something
that all ULP should be aware of and some go so far as to communicate that
as part of the Hello / login exchange. This allows the ULP
implementation to determine whether it wants to stall or wants to wait
until Read Responses complete before sending another request. This
isn't something silent; this isn't something new; this is something for
the ULP implementation to decide how to deal with the issue.
<br><br>
BTW, this is part of the hardware and associated specifications so it is
up to software to deal with the limited hardware resources and the
associated consequences. Please keep in mind that there are a
limited number of RDMA Request / Atomic resource "slots" at the
receiving HCA / RNIC. These are kept in hardware thus one must know
the exact limit to avoid creating protocol problems. A ULP
transmitter may post to the transmit queue more than the allotted slots
but the transmitting (source) HCA / RNIC must not issue them to the
remote. These requests do cause the source to stall. This is
a well understood problem and if people give the iSCSI / iSER and DA
specs good read or SDP they can see that this issue is
comprehended. I agree with people that ULP designers / implementers
must pay close attention to this constraint as it is in the iWARP / IB
specifications for a very good reason and these semantics must be
preserved to maintain the ordering requirements that are the used by the
overall RDMA protocols themselves.<br><br>
Mike<br><br>
<br><br>
At 05:24 AM 6/6/2006, Talpey, Thomas wrote:<br>
<blockquote type=cite class=cite cite="">At 03:43 AM 6/6/2006, Michael S.
Tsirkin wrote:<br>
>Quoting r. Talpey, Thomas <Thomas.Talpey@netapp.com>:<br>
>> Semantically, the provider is not required to provide any such
flow control<br>
>> behavior by the way. The Mellanox one apparently does, but it is
not<br>
>> a requirement of the verbs, it's a requirement on the upper
layer. If more<br>
>> RDMA Reads are posted than the remote peer supports, the
connection<br>
>> may break.<br>
><br>
>This does not sound right. Isn't this the meaning of this field:<br>
>"Initiator Depth: Number of RDMA Reads & atomic
operations<br>
>outstanding at any time"? Shouldn't any provider enforce this
limit?<br><br>
The core spec does not require it. An implementation *may* enforce
it,<br>
but is not *required* to do so. And as pointed out in the other
message,<br>
there are repercussions of doing so.<br><br>
I believe the silent queue stalling is a bit of a time bomb for upper
layers,<br>
whose implementers are quite likely unaware of the danger. I greatly<br>
prefer an implementation which simply sends the RDMA Read request,<br>
resulting in a failed (but unblocked!) connection. Silence is a very<br>
dangerous thing, no matter how helpful the intent.<br><br>
Tom.<br><br>
<br>
_______________________________________________<br>
openib-general mailing list<br>
openib-general@openib.org<br>
<a href="http://openib.org/mailman/listinfo/openib-general" eudora="autourl">
http://openib.org/mailman/listinfo/openib-general</a><br><br>
To unsubscribe, please visit
<a href="http://openib.org/mailman/listinfo/openib-general" eudora="autourl">
http://openib.org/mailman/listinfo/openib-general</a>
</font></blockquote></body>
</html>