[ofa-general] RDS flow control

[Olaf Kirch]

On Wednesday 14 May 2008 15:16:00 Olaf Kirch wrote:
> I'll let you know as soon as I have something for you to test.

Okay, here we go. I have a whole stack of patches sitting in
http://www.openfabrics.org/git/?p=~okir/ofed_1_3/linux-2.6.git
on branch future-20080516

This patch stack contains everything I'm working on right now, but
which isn't ready for OFED 1.3.1 yet (and I haven't started on 1.4
yet). So you get a lot more than you bargained for... but a fair
bit of that is actually needed because it prepares the ground for
the flow control stuff, so I didn't bother with ripping out the
unneeded pieces and rediffing everything.

I did some light testing with the code - it hasn't oopsed in a few hours,
but I'm getting occasional errors with rds-stress right after reloading
the module. They go away on the next attempt - so there's still something
fishy about the code (or about rds-stress).

I'm not completely happy with the performance yet. Early versions might
have been more adequately dubbed "trickle control" - but I eventually
managed to get something not-too-bad.

However, I'm still seeing performance degradation of ~5% with some packet
sizes. And that is *just* the overhead from exchanging the credit information
and checking it - at some point we need to take a spinlock, and that seems
to delay things just enough to make a dent in my throughput graph.

In fact, I haven't yet found a test case where the sender had to slow down
sending because it ran out of credits. Which confirms my suspicion that the
current setup isn't so bad, at least for IB...

If you're interested in the flow control code, the last commit
on that branch is the one to look at - commit header appended below.
I'm pretty sure that this is not exactly what iWARP needs, so please
send comments/patches on how to beat it into shape for iWARP.

Enjoyable weekend to everyone,
Olaf

commit e8a64b4f83df9df6617f75dff9e591b86174fa7c
Author: Olaf Kirch <olaf.kirch at oracle.com>
Date:   Fri May 16 06:16:40 2008 -0700

    RDS: Implement IB flow control

    Here it is - flow control for RDS/IB.

    This patch is still very much experimental. Here's the essentials

     -  The approach chosen here uses a credit-based flow control
        mechanism. Every SEND WR (including ACKs) consumes one credit,
        and if the sender runs out of credits, it stalls.

     -  As new receive buffers are posted, credits are transferred to the
        remote node (using yet another RDS header byte for this).

     -  Flow control is negotiated during connection setup. Initial credits
        are exchanged in the rds_ib_connect_private sruct - sending a value
        of zero (which is also the default for older protocol versions)
        means no flow control.

     -  We avoid deadlock (both nodes depleting their credits, and being
        unable to inform the peer of newly posted buffers) by requiring
        that the last credit can only be used if we're posting new credits
        to the peer.

    Flow control is configurable via sysctl. It only affects newly created
    connections however - so your best bet is to set this right after loading
    the RDS module.

    Signed-off-by: Olaf Kirch <olaf.kirch at oracle.com>


-- 
Olaf Kirch  |  --- o --- Nous sommes du soleil we love when we play
okir at lst.de |    / | \   sol.dhoop.naytheet.ah kin.ir.samse.qurax