[openib-general] client-server small message performance issues

[Roland Dreier]

 > Basic ping pong is 25 us.  That's fine as this is not a particularly
 > optimal way to communicate.  Each additional server adds 6 us.  That
 > seems like a lot of overhead just to do another pair of posts and
 > polls, but not my major complaint.  Look at the jump from 6 to 7
 > servers, 41 us.  Beyond that, too.  And the standard deviation
 > becomes huge.  A plot of the individual values shows a large spread,
 > not just a few outliers.

 > The hardware is all Mellanox MT25204

I would guess you are seeing the effect of exceeding the size of some
internal HCA cache, maybe the QP state cache.  But I don't know enough
details of the HCA internals to know if this is true and if so which
limit you're hitting.

 - R.