[openib-general][PATCH][kdapl]: evd upcall policy implementation

[Guy German]

Hi Caitlin,

Do you think that it is a good idea for the consumer
to continue reaping events from the EVD, while his
upcall policy is enabled  ?

Thanks,
Guy

Caitlin Bestler <mailto:caitlin.bestler at gmail.com> wrote:
> The DAT spec is fairly clear on this. The Consumer is responsible for
> either draining the EVD during the upcall, or for arranging
> for it to be
> drained at a later time through some out-of-scope mechanism.
> 
> The latter can include setting a flag that tells a background thread
> to continue reaping from the EVD when it is next scheduled.
> 
> On 8/15/05, James Lentini <jlentini at netapp.com> wrote:
>> 
>> 
>>>>>> You changed the order in which the CQ upcall is enabled and the
>>>>>> kDAPL upcall is made.
>>>> kDAPL doesn't dequeue the CQ events, so this shouldn't be an issue.
>>> 
>>> I think we agreed on that issue, on another thread, that
> I shall leave it:
>>>>  call kDAPL upcall
>>>>  enable CQ upcall
>>> right ?
>> 
>> Given that the ib_req_notify_cq() verb conforms to the IBTA spec's
>> Request Completion Notification verb, we will have to find another
>> solution. 
>> 
>>>> Currently the IB upcall is initially enabled, but there are checks
>>>> on the upcall path to determine if the EVD upcall is enabled. See
>>>> dapl_evd.c line 827: 
>>>> 
>>>>         if (state == DAPL_EVD_STATE_OPEN &&
>>>>             evd->upcall_policy != DAT_UPCALL_DISABLE) {
>>>> 
>>>> which you've replaced with
>>>> 
>>>>         if (state == DAPL_EVD_STATE_OPEN) {
>>> 
>>> Yes, because I believe the correct place to check it is inside
>>> dapl_evd_upcall_trigger, after the lock and before the actual
>>> upcall. 
>> 
>> I agree that checking it in dapl_evd_upcall_trigger() is better.
>> 
>> I still don't understand why you wouldn't setup the completion
>> notification based on the kDAPL consumer's upcall_policy in
>> dapl_evd_internal_create(). 
>> 
>>>>> You've only decrease the window in which that scenario could
>>>>> occur, not eliminated it. If a DTO completion occurred after you
>>>>> count the number of pending events but before you enable the CQ
>>>>> callback, a completion will be missed.
>>>>> 
>>>>> gg:
>>>>> I don't think so. That is what the spin_lock_irqsave is for.
>>>> 
>>>> What if there are multiple processors in the system?
>>> 
>>> AFAIK, spin_lock_irqsave does the trick. am I wrong ?
>> 
>> spin_lock_irqsave disables local interrupts.
>> 
>> In any event...[see question below]
>> 
>>>>> Also, the pending_event_queue is only used for kDAPL generated
>>>>> software events. This queue can be empty when there are events on
>>>>> the CQ, so your would need to be expanded your check to cover
>>>>> that. 
>>> 
>>> Actually, even though, I agreed before, I tend to disagree now.
>>> The consumer will still get the DTO events as soon as the CQ
>>> upcall is triggered (enabled), so only problem is with the pending
>>> events list.
>> 
>> Why is it an error for the consumer to modify the upcall policy when
>> there are pending events? 
>> 
>> dat_evd_modify_upcall should behave just like the IBTA spec's Request
>> Completion Notification verb in this respect. If there were events on
>> the EVD before the upcall is enabled, no upcall needs to be
>> generated. A correct consumer can easily work around this by
>> enabling the upcall and polling the EVD one final time to ensure it
>> is empty. _______________________________________________
>> openib-general mailing list
>> openib-general at openib.org
>> http://openib.org/mailman/listinfo/openib-general
>> 
>> To unsubscribe, please visit
> http://openib.org/mailman/listinfo/openib-general