[openib-general] Re: Mellanox HCAs: outstanding RDMAs

[Talpey, Thomas]

Todd, thanks for the set-up. I'm really glad we're having this discussion!

Let me give an NFS/RDMA example to illustrate why this upper layer,
at least, doesn't want the HCA doing its flow control, or resource
management.

NFS/RDMA is a credit-based protocol which allows many operations in
progress at the server. Let's say the client is currently running with
an RPC slot table of 100 requests (a typical value).

Of these requests, some workload-specific percentage will be reads,
writes, or metadata. All NFS operations consist of one send from
client to server, some number of RDMA writes (for NFS reads) or
RDMA reads (for NFS writes), then terminated with one send from
server to client.

The number of RDMA read or write operations per NFS op depends
on the amount of data being read or written, and also the memory
registration strategy in use on the client. The highest-performing
such strategy is an all-physical one, which results in one RDMA-able
segment per physical page. NFS r/w requests are, by default, 32KB,
or 8 pages typical. So, typically 8 RDMA requests (read or write) are
the result.

To illustrate, let's say the client is processing a multi-threaded
workload, with (say) 50% reads, 20% writes, and 30% metadata
such as lookup and getattr. A kernel build, for example. Therefore,
of our 100 active operations, 50 are reads for 32KB each, 20 are
writes of 32KB, and 30 are metadata (non-RDMA). 

To the server, this results in 100 requests, 100 replies, 400 RDMA
writes, and 160 RDMA Reads. Of course, these overlap heavily due
to the widely differing latency of each op and the highly distributed
arrival times. But, for the example this is a snapshot of current load.

The latency of the metadata operations is quite low, because lookup
and getattr are acting on what is effectively cached data. The reads
and writes however, are much longer, because they reference the
filesystem. When disk queues are deep, they can take many ms.

Imagine what happens if the client's IRD is 4 and the server ignores
its local ORD. As soon as a write begins execution, the server posts
8 RDMA Reads to fetch the client's write data. The first 4 RDMA Reads
are sent, the fifth stalls, and stalls the send queue! Even when three
RDMA Reads complete, the queue remains stalled, it doesn't unblock
until the fourth is done and all the RDMA Reads have been initiated.

But, what just happened to all the other server send traffic? All those
metadata replies, and other reads which completed? They're stuck,
waiting for that one write request. In my example, these number 99 NFS
ops, i.e. 654 WRs! All for one NFS write! The client operation stream
effectively became single threaded. What good is the "rapid initiation
of RDMA Reads" you describe in the face of this?

Yes, there are many arcane and resource-intensive ways around it.
But the simplest by far is to count the RDMA Reads outstanding, and
for the *upper layer* to honor ORD, not the HCA. Then, the send queue
never blocks, and the operation streams never loses parallelism. This
is what our NFS server does.

As to the depth of IRD, this is a different calculation, it's a DelayxBandwidth
of the RDMA Read stream. 4 is good for local, low latency connections.
But over a complicated switch infrastructure, or heaven forbid a dark fiber
long link, I guarantee it will cause a bottleneck. This isn't an issue except
for operations that care, but it is certainly detectable. I would like to see
if a pure RDMA Read stream can fully utilize a typical IB fabric, and how
much headroom an IRD of 4 provides. Not much, I predict.

Closing the connection if IRD is "insufficient to meet goals" isn't a good
answer, IMO. How does that benefit interoperability? 

Thanks for the opportunity to spout off again. Comments welcome!

Tom.

At 12:43 PM 6/6/2006, Rimmer, Todd wrote:
>
>
>> Talpey, Thomas
>> Sent: Tuesday, June 06, 2006 10:49 AM
>> 
>> At 10:40 AM 6/6/2006, Roland Dreier wrote:
>> >    Thomas> This is the difference between "may" and "must". The
>value
>> >    Thomas> is provided, but I don't see anything in the spec that
>> >    Thomas> makes a requirement on its enforcement. Table 107 says
>the
>> >    Thomas> consumer can query it, that's about as close as it
>> >    Thomas> comes. There's some discussion about CM exchange too.
>> >
>> >This seems like a very strained interpretation of the spec.  For
>> 
>> I don't see how strained has anything to do with it. It's not saying
>> anything
>> either way. So, a legal implementation can make either choice. We're
>> talking about the spec!
>> 
>> But, it really doesn't matter. The point is, an upper layer should be
>> paying
>> attention to the number of RDMA Reads it posts, or else suffer either
>the
>> queue-stalling or connection-failing consequences. Bad stuff either
>way.
>> 
>> Tom.
>
>Somewhere beneath this discussion is a bug in the application or IB
>stack.  I'm not sure which "may" in the spec you are referring to, but
>the "may"s I have found all are for cases where the responder might
>support only 1 outstanding request.  In all cases the negotiation
>protocol must be followed and the requestor is not allowed to exceed the
>negotiated limit.
>
>The mechanism should be:
>client queries its local HCA and determines responder resources (eg.
>number of concurrent outstanding RDMA reads on the wire from the remote
>end where this end will respond with the read data) and initiator depth
>(eg. number of concurrent outstanding RDMA reads which this end can
>initiate as the requestor).
>
>client puts the above information in the CM REQ.
>
>server similarly gets its information from its local CA and negotiates
>down the values to the MIN of each side (REP.InitiatorDepth =
>MIN(REQ.ResponderResources, server's local CAs Initiator depth);
>REP.ResponderResources = MIN(REQ.InitiatorDepth, server's local CAs
>responder resources).  If server does not support RDMA Reads, it can
>REJ.
>
>If client decided the negotiated values are insufficient to meet its
>goals, it can disconnect.
>
>Each side sets its QP parameters via modify QP appropriately.  Note they
>too will be mirror images of eachother:
>client:
>QP.Max RDMA Reads as Initiator = REP.ResponderResources
>QP.Max RDMA reads as responder = REP.InitiatorDepth
>
>server:
>QP.Max RDMA Reads as responder = REP.ResponderResources
>QP.Max RDMA reads as initiator = REP.InitiatorDepth
>
>We have done a lot of high stress RDMA Read traffic with Mellanox HCAs
>and provided the above negotiation is followed, we have seen no issues.
>Note however that by default a Mellanox HCA typically reports a large
>InitiatorDepth (128) and a modest ResponderResources (4-8).  Hence when
>I hear that Responder Resources must be grown to 128 for some
>application to reliably work, it implies the negotiation I outlined
>above is not being followed.
>
>Note that the ordering rules in table 76 of IBTA 1.2 show how reads and
>write on a send queue are ordered.  There are many cases where an op can
>pass an outstanding RDMA read, hence it is not always bad to queue extra
>RDMA reads.  If needed, the Fence can be sent to force order.
>
>For many apps, its going to be better to get the items onto queue and
>let the QP handle the outstanding reads cases rather than have the app
>add a level of queuing for this purpose.  Letting the HCA do the queuing
>will allow for a more rapid initiation of subsequent reads.
>
>Todd Rimmer