[openib-general][PATCH][kdapl]: evd upcall policy implementation

[Guy German]

Hi,

>> (It is possible to check the cq, before calling
>> dapl_evd_upcall_trigger, 
> 
> we could use ib_peek_cq
> 
>> for performance reasons, but applications that are interested
>> in performance, wouldn't go down that flow, anyway...)
> 
> You're assuming that applications interested in performance would not
> use upcalls? 

No. I'm assuming that applications interested in performance would disable 
the upcall policy (as soon as they receive the first upcall) and dequeue the
rest of the events from a thread. Therefore, when returning from 
dapl_evd_upcall_trigger - the upcall policy would be disabled and the 
"if" statement that follows it, would be false.

>>>>>> Also, the pending_event_queue is only used for kDAPL generated
>>>>>> software events. This queue can be empty when there are events on
>>>>>> the CQ, so your would need to be expanded your check to cover
>>>>>> that. 
>>>> 
>>>> Actually, even though, I agreed before, I tend to disagree now.
>>>> The consumer will still get the DTO events as soon as the CQ
>>>> upcall is triggered (enabled), so only problem is with the pending
>>>> events list.
>>> 
>>> Why is it an error for the consumer to modify the upcall policy
>>> when there are pending events? 
>>> 
>>> dat_evd_modify_upcall should behave just like the IBTA spec's
>>> Request Completion Notification verb in this respect. If there were
>>> events on the EVD before the upcall is enabled, no upcall needs to
>>> be generated. A correct consumer can easily work around this by
>>> enabling the upcall and polling the EVD one final time to ensure it
>>> is empty. 
>> 
>> There can be more than one event, and the consumer would need to
>> dequeue many times. While the consumer would do his extra dequeue-ing
>> he might also get an upcall, because his policy is now enabled.
>> I can't think of a design that can handle such a case, and if there
>> is one it is demanding and complicated, from the consumers side.
> 
> Isn't it the same position all event code written to the
> OpenIB API is
> in?

I don't quite know what you are reffering to, but if you are reffering
to the case of cq in IB - It's totally different: you only enable the cq
once, so you will only get one upcall, and the rest of the events
you will need to dequeue. 

> 
> I agree with you that this programming model is difficult to use,
> but I don't think it is impossible.

I think it is a bad idea to dequeue events and at the same time
receive upcalls from the same queue. It is racy, and has bad performance. 
I don't see *any* reason to do it.

Guy