[ofa-general] Non-Signaled RDMA Read (cxgb3)

[Roland Dreier]

 > I do not think this will help. When I do a send, I am expecting a
 > completion with the matching work request ID and I do not want to add
 > extra code into the upper layer app to get rid of those false
 > notifications (pitfall).

Your app would just get woken up, poll the CQ, and see that the CQ was
empty.  So it would presumably just go back to sleep until the next
event.

 > Speaking about this notification mechanism: What is the intended way
 > of using that?  Say I expect a number of inbound sends. What disturbs
 > me is that only the first of them generates an event. I cannot just
 > do an ibv_get_cq_event() after having gotten the first one.  It might
 > happen that already several inbound sends have arrived so I need to
 > poll the CQ for all of them. Only when the polling fails I go to
 > sleep waiting for an event. From a user perspective it would be way
 > more convenient to just call ibv_get_cq_event() several times.

CQ notification events are distinct from CQ entries.  events are fairly
heavyweight to create -- they involve servicing an interrupt from the
device and scheduling the right process to receive the event.  CQ
entries are cheap -- the adapter just DMAs a data structure into memory,
and the application can read it when it's ready.

 > Another disturbing thing is the fact that the events have to be
 > acknowledged - why bother the user with that?

There are races around destroying CQs and receiving events for CQs that
have already been destroyed that can't (AFAICS) be fixed without the
acknowledgement.  Note that you don't have to ack events until just
before you destroy the CQ.

 > If I may, I would like to suggest another thing: Is there no
 > possibility that non-signaled WRs do not create a WC at all? Clearing
 > the SQ with a signaled WR to prevent an overflow is rather cumbersome
 > as the user has to keep track of the number of unsignaled WRs posted
 > as well as the size of the SQ. Things like that add a lot of pitfalls
 > to upper layer applications.

If there are no completions ever then there is no way for the driver to
know when a send queue slot can be reused, and no way for the app to
know when resources can be released either.

 - R.