[openib-general][PATCH][kdapl]: evd upcall policy implementation

[James Lentini]

> >>> You changed the order in which the CQ upcall is enabled and the
> >>> kDAPL upcall is made. 
> > kDAPL doesn't dequeue the CQ events, so this shouldn't be an issue.
> 
> I think we agreed on that issue, on another thread, that I shall leave it:
> >  call kDAPL upcall
> >  enable CQ upcall
> right ?

Given that the ib_req_notify_cq() verb conforms to the IBTA spec's 
Request Completion Notification verb, we will have to find another 
solution.
 
> > Currently the IB upcall is initially enabled, but there are checks on
> > the upcall path to determine if the EVD upcall is enabled. See
> > dapl_evd.c line 827: 
> > 
> >         if (state == DAPL_EVD_STATE_OPEN &&
> >             evd->upcall_policy != DAT_UPCALL_DISABLE) {
> > 
> > which you've replaced with
> > 
> >         if (state == DAPL_EVD_STATE_OPEN) {
> 
> Yes, because I believe the correct place to check it is inside 
> dapl_evd_upcall_trigger, after the lock and before the actual upcall.

I agree that checking it in dapl_evd_upcall_trigger() is better. 

I still don't understand why you wouldn't setup the completion 
notification based on the kDAPL consumer's upcall_policy in 
dapl_evd_internal_create(). 

> >> You've only decrease the window in which that scenario could 
> >> occur, not eliminated it. If a DTO completion occurred after you 
> >> count the number of pending events but before you enable the CQ 
> >> callback, a completion will be missed.
> >> 
> >> gg:
> >> I don't think so. That is what the spin_lock_irqsave is for.
> > 
> > What if there are multiple processors in the system?
> 
> AFAIK, spin_lock_irqsave does the trick. am I wrong ?

spin_lock_irqsave disables local interrupts. 

In any event...[see question below]

> >> Also, the pending_event_queue is only used for kDAPL generated
> >> software events. This queue can be empty when there are events on the
> >> CQ, so your would need to be expanded your check to cover that.
> 
> Actually, even though, I agreed before, I tend to disagree now. 
> The consumer will still get the DTO events as soon as the CQ 
> upcall is triggered (enabled), so only problem is with the pending
> events list.

Why is it an error for the consumer to modify the upcall policy when 
there are pending events? 

dat_evd_modify_upcall should behave just like the IBTA spec's Request 
Completion Notification verb in this respect. If there were events on 
the EVD before the upcall is enabled, no upcall needs to be generated. 
A correct consumer can easily work around this by enabling the upcall 
and polling the EVD one final time to ensure it is empty.