[ewg] tcp_mem settings in /sbin/ib_ipoib_sysctl

[Richard Croucher]

The tcp memory  can only safely be set with some knowledge of the amount
of memory available to the kernel.   

I agree with Padeep that the default behaviour of letting this be set at
boot time is the safest approach.

Where necessary, and in specific cases, this can then be overridden by a
start up script.   

On Thu, 2010-12-02 at 12:06 -0800, Pradeep Satyanarayana wrote:

> Some customers have seen strange behaviors with the tcp_mem settings
> in /sbin/ib_ipoib_sysctl.The problems are similar to what has been
> reported at:
> 
> http://lists.linbit.com/pipermail/drbd-user/2009-September/012711.html
> 
> Essentially a setting of :
> /sbin/sysctl -q -w net.ipv4.tcp_mem="16777216 16777216 16777216"
> 
> chews up so much memory that it starves other applications (including
> file systems) and deadlocks the system. I am not sure which OFED
> release this crept in, but I 
> do understand that it is a performance tweak to help IPoIB CM. What
> may not have been considered is that this vector specifies pages (not
> bytes) as shown:
> 
> tcp_mem - vector of 3 INTEGERs: min, pressure, max
> min: below this number of pages TCP is not bothered about its
> memory appetite.
> 
> pressure: when amount of memory allocated by TCP exceeds this number
> of pages, TCP moderates its memory consumption and enters memory
> pressure mode, which is exited when memory consumption falls
> under "min".
> 
> max: number of pages allowed for queueing by all TCP sockets.
> 
> Defaults are calculated at boot time from amount of available
> memory.
> 
> In effect (with this setting)TCP will not moderate it's memory
> consumption below 16M*4K (if page size=4k) i.e. 64GB! This may be more
> than the RAM available on smaller
> systems, and what what about the case when page size is say 64K
> (1024GB before TCP starts to moderate)?
> 
> Can we consider removing the tcp_mem settings from
> the /sbin/ib_ipoib_sysctl file for OFED-1.5.3? As mentioned above,
> defaults are calculated at boot time
> based on the memory available and should be good enough for most uses.
> 
> Thanks
> Pradeep
> pradeep at us.ibm.com
> 
> 
> _______________________________________________
> ewg mailing list
> ewg at lists.openfabrics.org
> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/ewg/attachments/20101203/f0a98c45/attachment.html>