[openib-general] in need of a simple ulp

[Sean Hefty]

Bernhard Fischer wrote:
>>In this case you won't know what's on a cqe until you poll the completion. 
>>An alternative way of structuring the code is to create two CQs, and tie 
>>one to the send queue, the other to the receive queue.
> 
> hmz, yes, see "Quick unrelated question:" in
> http://openib.org/pipermail/openib-general/2005-May/006796.html
> 
> I do have ...s_cq and r_cq. For two nodes, using cqe=64 does obviously
> work, but when i've got, say, a setup where 786 nodes each have r and s?
> 
> This currently w{on't,ill in no way way} work (for me)!? Help!!
> 
> Also, Sean, any thought on the qp limit question referenced there?

The CQ and QP size limits are controlled by the HCA driver.  I don't see 
that mthca has any specific limitations, and would appear to be limited by 
system memory.

As a comparison, I was able to run tests with 1000 simultaneous connections, 
each transferring 1000 messages (send and receive QP sizes of 1000, and CQ 
size of 2000).  The message size was 1000, but I cheat and reuse the same 
data buffer, since I don't care about its actual content.

So, a cqe of 64 for 786 fully connected nodes seems doable, but will depend 
on how much space is allocated for the message buffers.

As a side note, when I tried increasing the number of connections to 5000 
under this same load, my system barfed with 'Out of Memory' errors...

- Sean