[ofw] [PATCH][RFC] winverbs/cq: implement CQ:Notify()

[Sean Hefty]

>What happens if a user has multiple Notify requests outstanding?  You only
>complete one?
>
>It seems that all should get completed.

I only complete one.  I'm not sure what the 'right' behavior should be here, but
completing all of them isn't difficult.  If there's a precedence here, I will
change the behavior.

>Also, why not provide richer status to distinguish overflow from catastrophic
>error?  Maybe something like STATUS_FAILURE for catastrophic failure, and
>STATUS_DATA_OVERRUN for CQ overflow?  Both are error status values.  Maybe it
>doesn't matter, though.

The only CQ event that I saw was IB_AE_CQ_ERROR.  I agree with providing richer
status, and do that with SRQ events.  Is there another async event that applies
to CQs?

>>+void WvCqNotify(WV_PROVIDER *pProvider, WDFREQUEST Request)
>>+{
>>+       WV_IO_ID                                *id;
>>+       WV_COMPLETION_QUEUE             *cq;
>>+       NTSTATUS                                status;
>>+       ib_api_status_t                 ib_status;
>>+
>>+       status = WdfRequestRetrieveInputBuffer(Request, sizeof(WV_IO_ID), &id,
>NULL);
>>+       if (!NT_SUCCESS(status)) {
>>+               goto out;
>>+       }
>>+
>>+       cq = WvCqAcquire(pProvider, id->Id);
>>+       if (cq == NULL) {
>>+               status = STATUS_NOT_FOUND;
>>+               goto out;
>>+       }
>>+
>>+       WdfObjectAcquireLock(cq->Queue);
>>+       ib_status = cq->pVerbs->enable_cq_notify(cq->hVerbsCq,
>>+
>id->Data ==
>>WV_IO_CQ_NOTIFY_SOLICITED);
>
>Are you sure this works - rearming a UM CQ in the kernel?  It should, but
>looking at the MTHCA code it appears that a doorbell for the CQ is only
>allocated in the kernel for kernel CQs (mthca_init_cq).
>
>How do you support a client that wants error notifications, but not rearm?

No, I'm not sure this works.  Let me think about the error notification versus
rearm; I may just need to extend WV_CQ_NOTIFY_TYPE to handle this.  My main
concern is ensuring fast userspace notification of completion events.  I may
need to call rearm from userspace after queuing the notification IOCTL in the
kernel.

- Sean