[openib-general] why sdp connections cost so much memory

[Michael S. Tsirkin]

Quoting r. zhu shi song <zhushisongzhu at yahoo.com>:
> Subject: why sdp connections cost so much memory
> 
> (1) ibv_devinfo
> HCA: MHES18-XTC 
> FW: 1.1.0
> OFED: OFED-1.1-rc1
> (2) Test Bed
> On Client:
> ib0: 193.12.10.24
> test command:
> LD_PRELOAD=/usr/local/ofed/lib64/libsdp.so
> SIMPLE_LIBSDP=1 ab -c m -n m -X 193.12.10.14:3129
> http://www.sse.com.cn/sseportal/ps/zhs/home.shtml
> The web page is about 68K.
> On Server:
> ib0: 193.12.10.14
> squid.sdp -d 10 -f squid2.conf (I have changed
> squid-cache to support listening on SDP port 3129)
> 
> The test result is :
> Concurrent Conns(=m)  Free Memory    Requests
> completed
> 0                         926980          0
> 100                       712508         100
> 200                       497372         200
> 300                       282636         256
> 400                       52868          256
> 500                       kernel crashed because of
> "out of memory"
> 
> >From above, every about 100 concurrent SDP connections
> will cost 210M memory.  It's too vast for large scale
> applications. TCP costs very lower memory than SDP. 
> The max concurrent connections completed successfully
> is 256. it is some bad limit.  Who knows how and when
> will solve the problem?
>   I'll test the performance of sdp connection and
> compare it with TCP further.
>    tks
>    zhu

Most memory in SDP goes into pre-posted receive buffers.
Currently SDP pre-posts a fixed 64 32K buffers per connection, that is
2M per connection.

To verify that's the issue, try opening drivers/infiniband/ulp/sdp/sdp.h
and changing SDP_RX_SIZE from 0x40 to a smaller value.
If this helps, as a quick work-around I can make this value
globally configurable.

TCP on the other hand scales down more gracefully, and so should
SDP longer-term.

> --- openib-general-request at openib.org wrote:
> 
> > Send openib-general mailing list submissions to
> > 	openib-general at openib.org
> > 
> > To subscribe or unsubscribe via the World Wide Web,
> > visit
> > 	http://openib.org/mailman/listinfo/openib-general
> > or, via email, send a message with subject or body
> > 'help' to
> > 	openib-general-request at openib.org
> > 
> > You can reach the person managing the list at
> > 	openib-general-owner at openib.org
> > 
> > When replying, please edit your Subject line so it
> > is more specific
> > than "Re: Contents of openib-general digest..."

Is this relevant somehow?

-- 
MST