[Openib-windows] RE: IPoIB RQ/SQ depth

[Fab Tillier]

Hi Yossi,

> From: Yossi Leybovich [mailto:sleybo at mellanox.co.il]
> Sent: Monday, December 05, 2005 12:24 AM
> 
> Hi
> 
> I tested the IPoIB with netperf, from my tests it shows that increasing
> the RQ/SQ depth from 128 to 512 add 10% in the bandwidth.
> With 128 I got 190 MB/s and with 512 I got 210 MB/s.(the test use 32k
> msg and 56k buffer in the socket)
> Is there any reason we not move the default depth to be 512 ?

I don't see any reason not to change the default values.  I hadn't had a chance
to do thorough testing so had left them alone.  Note that increasing the receive
queue depth likely has a larger effect, since receive processing is more CPU
intensive.  Do you get the same performance increase if you only change the
receive queue depth?  What about just increasing the ratio of the receive buffer
pool to RQ depth so that we have more receive buffers available?  It would be
great to find the best performing settings.

> Do you think that there is resource problem here ?

I don't think there's a resource problem at all.  Increasing the send queue
depth only increases the send WQE ring, nothing else.  On the receive side, the
impact is larger as the receive buffers are pre-allocated.

For In the environments IB is deployed in there will usually be plenty of
memory.  Users can always scale the value back using the adapter parameters if
they want to conserve resources.

- Fab