[ewg] [PATCH] mlx4_core: request MSIX vectors as much as there CPU cores

[Eli Cohen]

On Wed, May 05, 2010 at 12:55:54PM -0700, Roland Dreier wrote:
>  > We found it in performance work of our EN (10G) driver
> 
> By the way, it would certainly make sense for the ethernet driver to use
> a number of queues that matches num_online_cpus() at the time the
> interface is brought up.  Since we can't change the # of MSI-X vectors
> very easily I think we need to allow for the possible CPUs, but bouncing
> a net interface seems lighter weight to me.
> 
> Although perhaps reloading a driver on CPU hotplug is OK too?
> 

Yes, we have a system where num_possible_cpus is 32 and
num_online_cpus is 16. It's a RH5.4 and the kernel has no problem
allocating 33 MSI-X vectors. The point is that using more than one EQ
per CPU core does not buy us anything; in fact it can contiribute to a
higher rate of interrupts since the same EQ serves less CQs and the
chances for coalescing EQEs are lower.

So what do you think about the following patch to mlx4_en:


diff --git a/drivers/net/mlx4/en_cq.c b/drivers/net/mlx4/en_cq.c
index 21786ad..07c0779 100644
--- a/drivers/net/mlx4/en_cq.c
+++ b/drivers/net/mlx4/en_cq.c
@@ -49,11 +49,12 @@ int mlx4_en_create_cq(struct mlx4_en_priv *priv,
 {
 	struct mlx4_en_dev *mdev = priv->mdev;
 	int err;
+	int num_active_vectors = min_t(int, num_online_cpus(), mdev->dev->caps.num_comp_vectors);
 
 	cq->size = entries;
 	if (mode == RX) {
 		cq->buf_size = cq->size * sizeof(struct mlx4_cqe);
-		cq->vector   = ring % mdev->dev->caps.num_comp_vectors;
+		cq->vector   = ring % num_active_vectors;
 	} else {
 		cq->buf_size = sizeof(struct mlx4_cqe);
 		cq->vector   = 0;