[openib-general] different send and receive CQs

[Or Gerlitz]

Eric Barton wrote:
> In my ULP (lustre networking) I maintain a common pool of send descriptors and
> per-connection receive descriptors.  So it seems reasonable to have a single CQ
> for all sends and one CQ per-connection for receives.

Please note that since completions for each CQ make the HCA to generate 
an interrupt for which a SW handler needs some CPU to run on, multiple 
CQs scale upto the number of CPUs in the system. Beyond that your code 
will function quite bad. So its bad both for the client side (connecting 
to multiple OSTs and the server side connecting to $K-order clients).

An easy solution to the issue of a single CQ having RX completions from 
multiple connections (QPs) is to have the structure pointed by the 
cookie  carrying some TAG (pointer) relating it to the relevant connection.

some approaches for multiple CQs are:

+ separate RX and TX CQs
+ have multiple (#CPUs) threads being waken up by the CQ interrupt 
handler, where each (takes a lock...,) poll a completion and handles it.


Or.