[ofa-general] What should a ULP pass as ib_create_cq(..., comp_vector) ?

[Roland Dreier]

 > I notice the ib_create_cq() comp_vector support is merged in 2.6.22.
 > I don't completely understand what a ULP needs to pass as the argument.
 > 
 > I'm currently passing 0 in the NFS/RDMA client, what in general should I
 > consider using as a value? Or put another way, why is this exposed to
 > the ULP? Isn't this the MSI-X vector table index, a rather low-level thing
 > to hand to the ULP to manage?

You need to pass a value in the range 0 ... num_comp_vectors-1.  Since
every driver currently sets num_comp_vectors to 1, hard-coding your
value to 0 is a reasonable thing to do -- it's what every other ULP
does at the moment.

This value is *NOT* the MSI-X vector table index.  It's basically the
"completion event handler identifier" that the IB spec v. 1.2 talks
about.  It would be perfectly valid for a non-PCI device such as ehca
(for which the concept of MSI-X does not apply at all) to support
multiple completion vectors.  And the consumer is really the only
entity that can make a good choice of how to divide up CQs, since only
the consumer really knows which CQ event handlers might want to run in
parallel.

However on another level your question gets to the reason why we
haven't implemented support for multiple completion event vectors.
Namely, it's not clear how consumers, kernel or userspace, can make a
good choice of which vector to assign a given CQ to.  For example an
MPI implementation would probably want one vector per CPU so that it
can direct events for a given process to the CPU that the process is
running on; but there's no simple way to implement that policy.

 - R.