[ofa-general] virtual memory usage on long lived apps

[Steve Wise]

Hey Roland,

I have a verbs/memory mapping question for ya:

I've made some enhancements to the rdma-cm based 'rping' server to allow 
a persistent server mode.  This means the server process never exits, 
but rather spawns threads for each new incoming connection.  Like a 
traditional network server, except I'm using pthreads instead of new 
processes to handle each new connection.

Running this server with clients repeatedly connecting,  running rpings, 
and disconnecting, I notice that the virtual memory consumed by the 
process is ever increasing.  The shared/and reserved values don't 
increase, but the virtual does.  IE:  If you watch this process via 
'top' you'll see the VIRT value increasing steadily over 
time/connections, and the RES/SHR values stay steady.  I've seen it over 
1GB of virtual memory used!

Now, each new connection malloc()s memory for the rping test (around 
64KB), and registers this memory as part of setting up the new rping 
threads.  And I believe it properly unregisters and free()s this memory 
after the connection disconnects.

Q: Have you seen this before?  Its like the OS never reuses mappings 
that were setup for memory registrations.  Is that normal? Or should the 
virtual size shown in 'top' be reduced whenever a buffer is unregistered 
and freed?  There's a definite possibility the code is leaking memory, 
but I'm wondering if this is perhaps a normal anomaly with mapping 
memory in linux...

Thoughts?

Thanks,

Steve.