[openib-general][PATCH][kdapl]: evd upcall policy implementation

[Caitlin Bestler]

The DAT spec is fairly clear on this. The Consumer is responsible for
either draining the EVD during the upcall, or for arranging for it to be
drained at a later time through some out-of-scope mechanism.

The latter can include setting a flag that tells a background thread
to continue reaping from the EVD when it is next scheduled.

On 8/15/05, James Lentini <jlentini at netapp.com> wrote:
> 
> 
> > >>> You changed the order in which the CQ upcall is enabled and the
> > >>> kDAPL upcall is made.
> > > kDAPL doesn't dequeue the CQ events, so this shouldn't be an issue.
> >
> > I think we agreed on that issue, on another thread, that I shall leave it:
> > >  call kDAPL upcall
> > >  enable CQ upcall
> > right ?
> 
> Given that the ib_req_notify_cq() verb conforms to the IBTA spec's
> Request Completion Notification verb, we will have to find another
> solution.
> 
> > > Currently the IB upcall is initially enabled, but there are checks on
> > > the upcall path to determine if the EVD upcall is enabled. See
> > > dapl_evd.c line 827:
> > >
> > >         if (state == DAPL_EVD_STATE_OPEN &&
> > >             evd->upcall_policy != DAT_UPCALL_DISABLE) {
> > >
> > > which you've replaced with
> > >
> > >         if (state == DAPL_EVD_STATE_OPEN) {
> >
> > Yes, because I believe the correct place to check it is inside
> > dapl_evd_upcall_trigger, after the lock and before the actual upcall.
> 
> I agree that checking it in dapl_evd_upcall_trigger() is better.
> 
> I still don't understand why you wouldn't setup the completion
> notification based on the kDAPL consumer's upcall_policy in
> dapl_evd_internal_create().
> 
> > >> You've only decrease the window in which that scenario could
> > >> occur, not eliminated it. If a DTO completion occurred after you
> > >> count the number of pending events but before you enable the CQ
> > >> callback, a completion will be missed.
> > >>
> > >> gg:
> > >> I don't think so. That is what the spin_lock_irqsave is for.
> > >
> > > What if there are multiple processors in the system?
> >
> > AFAIK, spin_lock_irqsave does the trick. am I wrong ?
> 
> spin_lock_irqsave disables local interrupts.
> 
> In any event...[see question below]
> 
> > >> Also, the pending_event_queue is only used for kDAPL generated
> > >> software events. This queue can be empty when there are events on the
> > >> CQ, so your would need to be expanded your check to cover that.
> >
> > Actually, even though, I agreed before, I tend to disagree now.
> > The consumer will still get the DTO events as soon as the CQ
> > upcall is triggered (enabled), so only problem is with the pending
> > events list.
> 
> Why is it an error for the consumer to modify the upcall policy when
> there are pending events?
> 
> dat_evd_modify_upcall should behave just like the IBTA spec's Request
> Completion Notification verb in this respect. If there were events on
> the EVD before the upcall is enabled, no upcall needs to be generated.
> A correct consumer can easily work around this by enabling the upcall
> and polling the EVD one final time to ensure it is empty.
> _______________________________________________
> openib-general mailing list
> openib-general at openib.org
> http://openib.org/mailman/listinfo/openib-general
> 
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
>