[ofa-general] Re: Low NFS RDMA performance with Connect X

[Michael S. Tsirkin]

> Quoting James Lentini <jlentini at netapp.com>:
> Subject: RE: Low NFS RDMA performance with Connect X
> 
> 
> 
> On Wed, 5 Sep 2007, Kuchimanchi, Ramachandra wrote:
> 
> > John Leidel <john.leidel at gmail.com> wrote:
> >  
> > > In doing some testing with ConnectX, I noticed a similar issue in MPI
> > > performance.  The fix was simply to upgrade to the latetest and greatest
> > > firmware.
> > 
> > I tried with the latest ConnectX Firmware, version 2.2, and the Iozone
> > numbers are almost similar to what I posted previously and very low as
> > compared to the MT25208 numbers.
> > 
> > NFS RDMA folks, any ideas as to why this is happening with Connect X ?
> 
> We are bringing up our Connect X systems now (we're waiting on a 
> replacement memory dimm for our server). We'll be experimenting with 
> the performance on Connect X over the next few weeks.
> 
> Both the client and server code bases have been updated substantially 
> since the Mellanox SDK was released. Results are likely to be 
> different on the newer code.
> 
> Finally, it is conceivable that there will need to be performance 
> tweeks for the Connect X hardware. For Tavor hardware, ULPs use a 1KB 
> MTU to achieve maximum performance (see the setup of the path_mtu QP 
> attribute in net/sunrpc/xprtrdma/verbs.c).

One thing worth a try is interrupt coalescing.
The simplest way to check is probably to apply the following patch
and see if it helps. You can also try tweaking cq_max_count and
cq_period module parameters.

--->

From: Michael S. Tsirkin <mst at dev.mellanox.co.il>
Subject: [PATCH] IB/mlx4: enable interrupt coalescing

Enable interrupt coalescing for CQs in mlx4

Signed-off-by: Michael S. Tsirkin <mst at dev.mellanox.co.il>

---

diff --git a/drivers/net/mlx4/cq.c b/drivers/net/mlx4/cq.c
index 39253d0..b58dd75 100644
--- a/drivers/net/mlx4/cq.c
+++ b/drivers/net/mlx4/cq.c
@@ -42,6 +42,14 @@
 #include "mlx4.h"
 #include "icm.h"
 
+static int cq_max_count = 16;
+static int cq_period = 10;
+
+module_param(cq_max_count, int, 0444);
+MODULE_PARM_DESC(cq_max_count, "number of CQEs to generate event");
+module_param(cq_period, int, 0444);
+MODULE_PARM_DESC(cq_period, "time in usec for CQ event generation");
+
 struct mlx4_cq_context {
 	__be32			flags;
 	u16			reserved1[3];
@@ -174,6 +182,8 @@ int mlx4_cq_alloc(struct mlx4_dev *dev, int nent, struct mlx4_mtt *mtt,
 	cq_context->mtt_base_addr_h = mtt_addr >> 32;
 	cq_context->mtt_base_addr_l = cpu_to_be32(mtt_addr & 0xffffffff);
 	cq_context->db_rec_addr     = cpu_to_be64(db_rec);
+	cq_context->cq_max_count    = cq_max_count;
+	cq_context->cq_period       = cq_period;
 
 	err = mlx4_SW2HW_CQ(dev, mailbox, cq->cqn);
 	mlx4_free_cmd_mailbox(dev, mailbox);
-- 
MST