[ofa-general][PATCH] Re: mlx4: Completion EQ per cpu (MP support, Patch 10)

[Liran Liss]

> 
> I would just like to see an approach that is fully thought through and
> gives a way for applications/kernel drivers to choose a CQ vector
based
> on some information about what CPU it will go to.
>

Isn't the decision of which CPU an MSI-X is routed to (and hence, to
which CPI an EQ is bound to) determined by userspace? (either by the irq
balancer process or by manually setting /proc/irq/<vec>/smp_affinity)?

I am not sure we aren't better off leaving this to user-space: both
application and interrupt affinity are administrative tasks.
We can also use installation scripts to set a "default" configuration in
which vector 0 is bound to cpu0, vector 1 is bound to cpu1, etc.
 
> If we want to add a way to allow a request for round-robin, that is
> fine, but I don't think we want to change the default to round-robin,
> unless someone can come up with a workload where it actually helps.

Several IPoIB partitions can easily saturate a single core if their Rx
interrupts are not handled by several CPUs. This is not any different
from multiple Ethernet NICs whose interrupts are balanced today by the
irq balancer.

We can argue that IPoIB can use a special "round-robin" vector while
leaving the default vector fixed to a single EQ. However, there is
essentially no difference between IPoIB and other IB ULPs: an IB HCA is
actually a platform for other services, each with its own queues that
are directly accessed by HW, each with its own CQs and interrupt
moderation. Putting all these ULPs on a single EQ will prevent interrupt
balancing.

What are we risking in making the default action to spread interrupts?

--Liran