[openib-general][PATCH][kdapl]: evd upcall policy implementation

[James Lentini]

On Tue, 16 Aug 2005, Guy German wrote:

> >>>>>> Also, the pending_event_queue is only used for kDAPL generated
> >>>>>> software events. This queue can be empty when there are events on
> >>>>>> the CQ, so your would need to be expanded your check to cover
> >>>>>> that. 
> >>>> 
> >>>> Actually, even though, I agreed before, I tend to disagree now.
> >>>> The consumer will still get the DTO events as soon as the CQ
> >>>> upcall is triggered (enabled), so only problem is with the pending
> >>>> events list.
> >>> 
> >>> Why is it an error for the consumer to modify the upcall policy
> >>> when there are pending events? 
> >>> 
> >>> dat_evd_modify_upcall should behave just like the IBTA spec's
> >>> Request Completion Notification verb in this respect. If there were
> >>> events on the EVD before the upcall is enabled, no upcall needs to
> >>> be generated. A correct consumer can easily work around this by
> >>> enabling the upcall and polling the EVD one final time to ensure it
> >>> is empty. 
> >> 
> >> There can be more than one event, and the consumer would need to
> >> dequeue many times. While the consumer would do his extra dequeue-ing
> >> he might also get an upcall, because his policy is now enabled.
> >> I can't think of a design that can handle such a case, and if there
> >> is one it is demanding and complicated, from the consumers side.
> > 
> > Isn't it the same position all event code written to the
> > OpenIB API is
> > in?
> 
> I don't quite know what you are reffering to, but if you are reffering
> to the case of cq in IB - It's totally different: you only enable the cq
> once, so you will only get one upcall, and the rest of the events
> you will need to dequeue. 

The consumer should only receive one upcall at a time if the upcall 
policy is DAT_UPCALL_SINGLE_INSTANCE. If the dequeues are performed in 
an upcall, the logic needed in an OpenIB consumer and kDAPL consumer 
is essentially the same.

The difference is that the OpenIB consumer needs to re-enable the CQ 
upcall and poll to make sure no events were missed.

> > I agree with you that this programming model is difficult to use,
> > but I don't think it is impossible.
> 
> I think it is a bad idea to dequeue events and at the same time
> receive upcalls from the same queue. It is racy, and has bad performance. 
> I don't see *any* reason to do it.

The current kDAPL implementation does create a situation in which an 
upcall and poll occur simultaneously if the upcall is disabled, the 
consumer enables the upcall, and then the consumer does a poll. In 
this scenario an upcall can occur while the consumer is polling. I was 
pointing out that this same race exists in the OpenIB verbs API (and 
the IBTA verbs).

Again, I agree that we can eliminate the additional poll after 
enabling the upcall in kDAPL. We just need to do it in a way that is 
not hardware specific. I believe we can use the same technique we did 
in the DTO upcall.

james