[ofw] receive queue depth effect on pingpong latency

[Tzachi Dar]

Hi Fab,

I'm sorry that I don't have time to look at this thoroughly this week (maybe someone else will).

In any case, this looks to me like "End of queue" for IB.

What this means is that if there are no receive wqes, the card sends nacks and stops traffic. (probably not your case)
When there is only one receive packet IB also does not work efficiency, and fw flow is being used. (this is probably what you see ).

This does not explain everything that you see, but it probably explains the first 3 lines.

By the way, can you post more receive packets and see if this helps?

Thanks
Tzachi

> -----Original Message-----
> From: Fab Tillier [mailto:ftillier at microsoft.com]
> Sent: Monday, October 18, 2010 10:07 PM
> To: Tzachi Dar; Leonid Keller
> Cc: ofw at lists.openfabrics.org
> Subject: receive queue depth effect on pingpong latency
> 
> Hi Tzachi, Leo,
> 
> I've been playing with the ndpingpong test case, and noticed some
> strange/unexpected behavior:
> 
> What I see is a relationship between the RQ depth and the latency where
> the larger the RQ depth, the lower the latency.  This, despite the
> program performing a pingpong: a send is only issued once a receive is
> completed, so there should only be a single work request in transit at
> a time.
> 
> I changed the unit test to use an asymmetric queue depth, keeping the
> SQ depth at 1, and varying only the RQ depth.
> 
> Here are the reported results for the 1 byte message size, by RQ depth:
> 
> RQ 1: 7.44us
> RQ 2: 4.76us
> RQ 4: 3.20us
> RQ 6 (default): 2.75us
> RQ 8: 2.44us
> RQ 16: 2.04us
> RQ 32: 1.85us
> RQ 64: 1.76us
> RQ 128: 1.71us
> RQ 256: 1.71us
> RQ 512: 1.68us
> RQ 1024: 1.68us
> RQ 2048: 1.67us
> RQ 4096: 1.67us
> 
> As you can see, things are reaching steady state as the queue depth
> gets very large.  But as this is a ping pong test, I would have
> expected performance to be much closer to this for the smaller message
> sizes.
> 
> This is with ConnectX2, QDR, FW 2.07.9110
> 
> Any idea why the low RQ depth tests perform so poorly?
> 
> Thanks,
> -Fab