[ewg] tcp_mem settings in /sbin/ib_ipoib_sysctl

[Pradeep Satyanarayana]

Some customers have seen strange behaviors with the tcp_mem settings
in /sbin/ib_ipoib_sysctl.The problems are similar to what has been reported
at:

http://lists.linbit.com/pipermail/drbd-user/2009-September/012711.html

Essentially a setting of :
/sbin/sysctl -q -w net.ipv4.tcp_mem="16777216 16777216 16777216"

chews up so much memory that it starves other applications (including file
systems) and deadlocks the system. I am not sure which OFED release this
crept in, but I
do understand that it is a performance tweak to help IPoIB CM. What may not
have been considered is that this vector specifies pages (not bytes) as
shown:

tcp_mem - vector of 3 INTEGERs: min, pressure, max
        min: below this number of pages TCP is not bothered about its
        memory appetite.

        pressure: when amount of memory allocated by TCP exceeds this
number
        of pages, TCP moderates its memory consumption and enters memory
        pressure mode, which is exited when memory consumption falls
        under "min".

        max: number of pages allowed for queueing by all TCP sockets.

        Defaults are calculated at boot time from amount of available
        memory.

In effect (with this setting)TCP will not moderate it's memory consumption
below 16M*4K (if page size=4k) i.e. 64GB! This may be more than the RAM
available on smaller
systems, and what what about the case when page size is say 64K (1024GB
before TCP starts to moderate)?

Can we consider removing the tcp_mem settings from
the /sbin/ib_ipoib_sysctl file for OFED-1.5.3? As mentioned above, defaults
are calculated at boot time
based on the memory available and should be good enough for most uses.

Thanks
Pradeep
pradeep at us.ibm.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/ewg/attachments/20101202/bf228ba7/attachment.html>