[ofa-general] QP needs a lot of memory

[Steve Wise]

Sean Hefty wrote:
>> I am running OFED 1.4 on a Chelsio T3 RNIC.
>> When I was trying to connect a large number of clients (several hundred),
>> I noticed that the server was running out of memory. One instance of a
>> whenever I do an rdma_create_qp(), I lose about 7MB of main memory.
>> This severly limits the scalability of my application.
>>
>> Is there a reason for that?
>>     
>
> The amount of memory needed per QP is based on the send and receive queue sizes,
> plus the number of SGEs.  I don't know specific details about the Chelsio
> adapter itself to know if 7MB is high or not.
>
>   
That seems very high.  A T3 max depth QP should only use around 256KB of 
dma coherent memory.   The max depth CQ should be around 256KB too.  So 
something whacked if its consuming 7MB per QP...

How are you measuring this memory usage?


Steve.