[ofa-general] QP needs a lot of memory

[Philip Frey1]

> >> I am running OFED 1.4 on a Chelsio T3 RNIC.
> >> When I was trying to connect a large number of clients (several 
hundred),
> >> I noticed that the server was running out of memory. One instance of 
a
> >> whenever I do an rdma_create_qp(), I lose about 7MB of main memory.
> >> This severly limits the scalability of my application.
> >>
> >> Is there a reason for that?
> >> 
> >
> > The amount of memory needed per QP is based on the send and 
> receive queue sizes,
> > plus the number of SGEs.  I don't know specific details about the 
Chelsio
> > adapter itself to know if 7MB is high or not.
> >
> > 
> That seems very high.  A T3 max depth QP should only use around 256KB of 

> dma coherent memory.   The max depth CQ should be around 256KB too.  So 
> something whacked if its consuming 7MB per QP...
> 
> How are you measuring this memory usage?

I measured it using the vmstat command. The 7MB are only a rough 
estimation.
Since I was not aware of the fact that the size of a QP object is 
depending on
the max QP init attrs, I assigned the highest possible values for each
(16384 send WRs, 1023 recv WRs, 4 SGEs for send and recv each). When 
reducing
these numbers to what is actually needed (2 send WRs, 2 recv WRs and 1 SGE 
each),
the QP size shrank significantly and I was able to create many more QPs.

Thanks,
 Philip
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20090121/a43b193e/attachment.html>