[ofa-general] missed cq event

[Pete Wyckoff]

rdreier at cisco.com wrote on Tue, 10 Jun 2008 12:01 -0700:
>  > You know, I've never quite liked this fact.  I've seen multiple
>  > people, including myself, get tripped up by this issue.  And now we
>  > have lots of applications with code to:  determine the queue depth,
>  > increment a counter on every send and periodically post a SIGNALED
>  > one, and handle the uninteresting CQ event.
>  > 
>  > Trying not to think about how the hardware works for a moment, this
>  > issue can be seen as yet another little detail that apps people must
>  > learn to use RDMA devices.  Any ideas on how verbs could handle this
>  > case itself?
> 
> I don't see a good way that this could be handled at the verbs level.
> How does the app know when the resources associated with the send
> request can be freed?  How would you size your CQ (given than any and
> all unsignaled requests might generate a completion with error)?
> 
> What would you want to happen if an app posts 101 work requests to a
> queue of depth 100 before any WRs complete?
> 
> It seems to me that the app needs the periodic completion just as much
> as the hardware driver needs it for book-keeping.

Those are valid comments in general.  My thoughts are coming from a
particular application mindset:  for situations where you need
app-level flow control anyway.  One example is RPC, where a send
going out will always generate a receive coming back in.  Remote
storage or file systems are a case where we need flow control anyway
to avoid overflowing various queueing points in the server, so it is
easy to piggy-back freeing the send resources when the write ack or
read result comes back.  It's not possible to free the send
resources before seeing the app-level ack from the other side;
something can go wrong even after the send CQ event happens.

CQ depth can be chosen based on the maximum number of outstanding
file system requests as dictated by the server, as we still need the
space for errors as you point out.  Outstanding WR limit still
applies as a separate per-QP limit and is also sizeable based on app
flow control constraints.

You're certainly right that the need to reap even unsignaled sends
is needed for some apps.  I'm just looking for a shortcut if my app
does not ever need to find out about send completions.  Just to
simplify the code, not expecting anything will go faster or require
less memory.

		-- Pete