[openib-general] why sdp connections cost so much memory

[zhu shi song]

I have changed SDP_RX_SIZE from 0x40 to 1 and rebuilt
ib_sdp.ko. But kernel always crashed.
zhu

--- "Michael S. Tsirkin" <mst at mellanox.co.il> wrote:

> Quoting r. zhu shi song <zhushisongzhu at yahoo.com>:
> > Subject: why sdp connections cost so much memory
> > 
> > (1) ibv_devinfo
> > HCA: MHES18-XTC 
> > FW: 1.1.0
> > OFED: OFED-1.1-rc1
> > (2) Test Bed
> > On Client:
> > ib0: 193.12.10.24
> > test command:
> > LD_PRELOAD=/usr/local/ofed/lib64/libsdp.so
> > SIMPLE_LIBSDP=1 ab -c m -n m -X 193.12.10.14:3129
> > http://www.sse.com.cn/sseportal/ps/zhs/home.shtml
> > The web page is about 68K.
> > On Server:
> > ib0: 193.12.10.14
> > squid.sdp -d 10 -f squid2.conf (I have changed
> > squid-cache to support listening on SDP port 3129)
> > 
> > The test result is :
> > Concurrent Conns(=m)  Free Memory    Requests
> > completed
> > 0                         926980          0
> > 100                       712508         100
> > 200                       497372         200
> > 300                       282636         256
> > 400                       52868          256
> > 500                       kernel crashed because
> of
> > "out of memory"
> > 
> > >From above, every about 100 concurrent SDP
> connections
> > will cost 210M memory.  It's too vast for large
> scale
> > applications. TCP costs very lower memory than
> SDP. 
> > The max concurrent connections completed
> successfully
> > is 256. it is some bad limit.  Who knows how and
> when
> > will solve the problem?
> >   I'll test the performance of sdp connection and
> > compare it with TCP further.
> >    tks
> >    zhu
> 
> Most memory in SDP goes into pre-posted receive
> buffers.
> Currently SDP pre-posts a fixed 64 32K buffers per
> connection, that is
> 2M per connection.
> 
> To verify that's the issue, try opening
> drivers/infiniband/ulp/sdp/sdp.h
> and changing SDP_RX_SIZE from 0x40 to a smaller
> value.
> If this helps, as a quick work-around I can make
> this value
> globally configurable.
> 
> TCP on the other hand scales down more gracefully,
> and so should
> SDP longer-term.
> 
> > --- openib-general-request at openib.org wrote:
> > 
> > > Send openib-general mailing list submissions to
> > > 	openib-general at openib.org
> > > 
> > > To subscribe or unsubscribe via the World Wide
> Web,
> > > visit
> > > 
> http://openib.org/mailman/listinfo/openib-general
> > > or, via email, send a message with subject or
> body
> > > 'help' to
> > > 	openib-general-request at openib.org
> > > 
> > > You can reach the person managing the list at
> > > 	openib-general-owner at openib.org
> > > 
> > > When replying, please edit your Subject line so
> it
> > > is more specific
> > > than "Re: Contents of openib-general digest..."
> 
> Is this relevant somehow?
> 
> -- 
> MST
> 


__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com