Selective signalling vs WQE size - was Re: [ofa-general] mthca max_sge value... ugh.

[Roland Dreier]

 > >if we can't use the "WQE shrinking" feature (because of selective
 > >signaling in the NFS/RDMA case), and we want to use 32 sge entries, then
 > >the WQE size 's' will end up a little more than 512 bytes, and the
 > >wqe_shift will end up as 10.
 > 
 > Can you elaborate on this? The NFS/RDMA client does selective signalling
 > on its send queue in order to save on interrupts and CQE generation/handling.
 > Which I always thought was a (very) good approach. Because the RPC
 > request/response paradigm guarantees an eventual receive completion,
 > we simply defer (or even completely avoid) this work.
 > 
 > Would that be a bad trade if it takes a WQE management opportunity away
 > from the provider? It's quite easy to change this in the NFS/RDMA code,
 > or make it a selectable parameter.

mlx4 has a feature that lets the driver post smaller WQEs to the send
queue if not all s/g entries are used.  But the current implementation
at least can only use this feature if selective signaling is off.

So it's a tradeoff -- more work completions or bigger data structures
for the HCA to fetch.  In the NFS/RDMA case I would expect the selective
signaling to be a win, but I guess the thing to do is try ConnectX
without selective signaling and see which wins.

 - R.