[openib-general] client-server small message performance issues

[Pete Wyckoff]

rdreier at cisco.com wrote on Tue, 17 Oct 2006 14:24 -0700:
>  > Basic ping pong is 25 us.  That's fine as this is not a particularly
>  > optimal way to communicate.  Each additional server adds 6 us.  That
>  > seems like a lot of overhead just to do another pair of posts and
>  > polls, but not my major complaint.  Look at the jump from 6 to 7
>  > servers, 41 us.  Beyond that, too.  And the standard deviation
>  > becomes huge.  A plot of the individual values shows a large spread,
>  > not just a few outliers.
> 
>  > The hardware is all Mellanox MT25204
> 
> I would guess you are seeing the effect of exceeding the size of some
> internal HCA cache, maybe the QP state cache.  But I don't know enough
> details of the HCA internals to know if this is true and if so which
> limit you're hitting.

Mellanox picked up on my email and sent me new firmware that
contains some optimizations for that particular silicon.  The
pre-release firmware image makes the numbers look much more
reasonable.

		-- Pete