[ofa-general] IPOIB CM (NOSRQ) patch -memory footprint

[Pradeep Satyanarayana]

Roland Dreier wrote:
>  > By default, cap the NOSRQ memory usage to 1GB. The default recvq_size
>  > is set to 128. Therefore for 64KB packets this would imply a maximum of
>  > 128 endpoints.
> 
> 1 GB is a pretty eye-popping amount to tie up in receive buffers.

It is 8MB per endpoint of receive buffers, and so you would use up 1GB
when all 128 endpoints are active at the same time.

This proposal is only for PPC systems when used with IBM HCA. It has no
impact on Topspin HCAs (even when used on PPC systems).

IBM cluster nodes are "fat" systems and have large quantities of memory.
Hence using up 1GB should not be an issue.

> 
>  > -The NOSRQ limit of 1GB is also made a module parameter.
> 
> There are too many module parameters already I think...

If we choose the defaults correctly, most customers should not have to
tune these parameters. This way we provide the flexibility to systems
with large memory (say 64 GB) to raise the limits if need be.


> 
>  > -Currently we allocate a default of 64KB for the ring buffer elements,
>  > and this buffer size is not linked to the mtu. In the future, we could
>  > allocate buffers based on the mtu and link that into the computation of
>  > the memory cap. This way customers who might want to use a smaller mtu
>  > could use a larger number of endpoints, or a larger recvq_size without
>  > exceeding the memory cap.
> 
> It would be nice, but to handle increasing the MTU, you need some way
> to handle the receives you already posted (which would be too small
> all of a sudden).
> 
Can you expand on this a little more -I do not catch the drift.

Pradeep