[ofw][patch][ND provider] Improving latency of ms-mpi

[Sean Hefty]

>Why would all sends have to be the same size?  The inline tradeoff is between
>writing a 16-byte data segment, and then doing a DMA of the data, vs. copying
>the data direct to the SGE.  It shouldn't matter if the sends are all the same
>size.  There's a point where doing the copy is more efficient than setting up
>the data segment.

If you set max inline to 400, but do 16 byte transfers, that's worse than
setting max inline to 16.  There's more to the cost of having a larger max
inline value than copying versus registering memory.  This is a property of the
application, not the hardware.

You need separate values for placing the data directly into the SGL, versus
avoiding a registration.

>It's like passing data by value or by reference.

If you add that the function should always take 100 parameters, then I'll agree.
:)

- Sean