[ofa-general] OOM problem with ib_ipoib?

[John Marshall]

Roland Dreier wrote:
>> The curious thing is that the OOM occurs even when the ib interfaces
>> are _not even UP_, although the ib_ipoib module is loaded. So, I cannot
>> see how it can be an allocation issue in such a case related to usage. Am I
>> missing something here?
>>     
>
> The IPoIB CM code allocates receive buffers even before the interface is brought
> up.  Maybe the wrong thing to do, but that's how the code is now at least.
>
>   
>> As well, shouldn't the OS handle this transparently via the pdflush which
>> will write out the data and free up memory? Or does the pdflush not
>> distinguish between total memory and low memory so that a problem
>> occurs (yet the OOM happens even when the interfaces are not UP!)?
>>     
>
> You may really have no free lowmem... keep in mind that the linux mm really
> does not behave well with 32G of RAM and a 32-bit kernel.  It's fundamentally
> and insane config and so no one tunes for it.
>   
Progress!

1) I have done further tests and am comfortable that they do not happen on
the x86-64 platform.

2) More tests using the same equipment but again with bigmem and, given
your pointer on lowmem, have found that if I tweak the system with sysctl
setting of:
    vm.lowmem_reserve_ratio=128 128 32
things seem to work well. I do this on _both_ the server and the client 
sides
(lowmem issues also pop up on the client side when using nfs).

Thanks,
John