[ofa-general][PATCH] Re: mlx4: Completion EQ per cpu (MP support, Patch 10)

[Roland Dreier]

 > > I would just like to see an approach that is fully thought through and
 > > gives a way for applications/kernel drivers to choose a CQ vector based
 > > on some information about what CPU it will go to.

 > Isn't the decision of which CPU an MSI-X is routed to (and hence, to
 > which CPI an EQ is bound to) determined by userspace? (either by the irq
 > balancer process or by manually setting /proc/irq/<vec>/smp_affinity)?

Yes, but how can anything tell which IRQ number corresponds to a given
"CQ vector" number?  (And don't be too stuck on MSI-X, since ehca uses
some completely different GX-bus related thing to get multiple interrupts)

 > What are we risking in making the default action to spread interrupts?

There are fairly plausible scenarios like a multi-threaded app where
each thread creates a send CQ and a receive CQ, which should both be
bound to the same CPU as the thread.  If we spread all CQs then it's
impossible to get thread-locality.

I'm not saying that round-robin is necessarily a bad default policy, but
I do think there needs to be a complete picture of how that policy can
be overridden before we go for multiple interrupt vectors.

 - R.