<html>
<body>
<font size=3>At 03:33 PM 9/21/2005, Caitlin Bestler wrote:<br>
<blockquote type=cite class=cite cite="">I'm not sure I follow what a
"completion channel" is.<br>
My understanding is that work completions are stored in<br>
user-accessible memory (typically a ring buffer). This <br>
enables fast-path reaping of work completions. The OS<br>
has no involvement unless notifications are enabled.<br><br>
The "completion vector" is used to report completion<br>
notifications. So is the completion vector a *single*<br>
resource used by the driver/verbs to report completions,<br>
where said notifications are then split into user<br>
context dependent "completion channels"?<br><br>
The RDMAC verbs did not define callbacks to userspace<br>
at all. Instead it is assumed that the proxy for user<br>
mode services will receive the callbacks, and how it<br>
relays those notifications to userspace is outside<br>
the scope of the verbs.</font></blockquote><br>
Correct.<br><br>
<br>
<blockquote type=cite class=cite cite=""><font size=3>Both uDAPL and
ITAPI define relays of notifications<br>
to AEVDS/CNOs and/or file descriptors. Forwarding<br>
a completion notification to userspace in order to<br>
make a callback in userspace so that it can kick<br>
an fd to wake up another thread doesn't make much<br>
sense. The uDAPL/ITAPI/whatever proxy can perform<br>
all of these functions without any device dependencies<br>
and in a way that is fully optimal for the usermode<br>
API that is being used. </font></blockquote><br>
Exactly. This was the intention. Does not really matter what the
API is but that there by an API that does this work on behalf of the
consumer.<br><br>
<blockquote type=cite class=cite cite=""><font size=3>For kernel clients,
I don't see any need for anything beyond the already defined<br>
callbacks direct from the device-dependent code.</font></blockquote><br>
This was the intention when we designed the verbs.<br><br>
<blockquote type=cite class=cite cite=""><font size=3>Even in the typical
case where the usermode application<br>
does an evd_wait() on the DAT or ITAPI endpoint, the<br>
DAT/ITAPI proxy will be able to determine which thread<br>
should be woken and could even do so optimally. It<br>
also allows the proxy to implemenet Access Layer features<br>
such as EVD thresholding without device-specific support.
</font></blockquote><br>
Correct.<br><br>
<br>
<blockquote type=cite class=cite cite=""><font size=3>> -----Original
Message-----<br>
> From: openib-general-bounces@openib.org <br>
>
[<a href="mailto:openib-general-bounces@openib.org" eudora="autourl">
mailto:openib-general-bounces@openib.org</a>] On Behalf Of Roland
Dreier<br>
> Sent: Wednesday, September 21, 2005 12:22 PM<br>
> To: openib-general@openib.org<br>
> Subject: [openib-general] [RFC] libibverbs completion event
handling<br>
> <br>
> While thinking about how to handle some of the issues raised <br>
> by Al Viro in
<<a href="http://lkml.org/lkml/2005/9/16/146" eudora="autourl">
http://lkml.org/lkml/2005/9/16/146</a>>, I <br>
> realized that our verbs interface could be improved to make <br>
> delivery of completion events more flexible. For example,
<br>
> Arlin's request for using one FD for each CQ can be <br>
> accomodated quite nicely.<br>
> <br>
> The basic idea is to create new objects that I call <br>
> "completion vectors" and "completion
channels." Completion <br>
> vectors refer to the interrupt generated when a completion <br>
> event occurs. With the current drivers, there will always be
<br>
> a single completion vector, but once we have full MSI-X <br>
> support, multiple completion vectors will be
possible.</font></blockquote><br>
When I proposed the use of multiple completion handlers, it was based on
the operating assumption that either MSI or MSI-X be used by the
underlying hardware. Either is possible - MSI limits it to a single
address with 32 data values which allows different handlers to be bound
to each value though targeting a single processor. MSI-X builds
upon technology we've been shipping for nearly 20 years now and allows up
to 2048 different addresses which may target or multiple
processors. Any API should be able to deal with both approaches
thus should not assume anything about whether one or more handlers are
bound to a given processor.<br><br>
<blockquote type=cite class=cite cite=""><font size=3>> Orthogonal to
this is the notion of a completion channel. <br>
> This is a FD used for delivering completion events to
userspace.<br>
> <br>
> Completion vectors are handled by the kernel, and userspace <br>
> cannot change the number of vectors that available. On the
<br>
> other hand, completion channels are created at the request of <br>
> a userspace process, and userspace can create as many <br>
> channels as it wants.<br>
> <br>
> Every userspace CQ has a completion vector and a completion
channel.<br>
> Multiple CQs can share the same completion vector and/or the <br>
> same completion channel. CQs with different completion <br>
> vectors can still share a completion channel, and vice versa.<br>
> <br>
> The exact API would be something like the below.
Thoughts?</font></blockquote><br>
Why wouldn't it just be akin to the verbs interface - here are the event
handler and callback routines to associate with a given CQ. The
handler might be nothing more than an index into a set of functions that
are stored within the kernel - these functions are either device-specific
(i.e. supplied by the IHV) or a OS-specific such as dealing with error
events (might also have a device-specific component as well). When
the routine is invoked, it has basically has three parameters: CQ to
target, number of CQE to reap, address to store CQE. I do not see
what more is required.<br><br>
Mike<br><br>
<blockquote type=cite class=cite cite=""><font size=3>> <br>
> Thanks,<br>
> Roland<br>
> <br>
> struct ibv_comp_channel {<br>
>
<x-tab> </x-tab>int<x-tab>
</x-tab><x-tab>
</x-tab><x-tab>
</x-tab>fd;<br>
> };<br>
> <br>
> /**<br>
> * ibv_create_comp_channel - Create a completion event <br>
> channel */ extern struct ibv_comp_channel <br>
> *ibv_create_comp_channel(struct ibv_context *context);<br>
> <br>
> /**<br>
> * ibv_destroy_comp_channel - Destroy a completion event <br>
> channel */ extern int ibv_destroy_comp_channel(struct <br>
> ibv_comp_channel *channel);<br>
> <br>
> /**<br>
> * ibv_create_cq - Create a completion queue<br>
> * @context - Context CQ will be attached to<br>
> * @cqe - Minimum number of entries required for CQ<br>
> * @cq_context - Consumer-supplied context returned for <br>
> completion events<br>
> * @channel - Completion channel where completion events will
<br>
> be queued.<br>
> * May be NULL if completion events
will not be used.<br>
> * @comp_vector - Completion vector used to signal completion
events.<br>
> * Must be >= 0 and <
context->num_comp_vectors.<br>
> */<br>
> extern struct ibv_cq *ibv_create_cq(struct ibv_context <br>
> *context, int cqe,<br>
>
<x-tab> </x-tab><x-tab>
</x-tab><x-tab>
</x-tab><x-tab>
</x-tab>
void *cq_context,<br>
>
<x-tab> </x-tab><x-tab>
</x-tab><x-tab>
</x-tab><x-tab>
</x-tab>
struct ibv_comp_channel *channel,<br>
>
<x-tab> </x-tab><x-tab>
</x-tab><x-tab>
</x-tab><x-tab>
</x-tab>
int comp_vector);<br>
> <br>
> /**<br>
> * ibv_get_cq_event - Read next CQ event<br>
> * @channel: Channel to get next event from.<br>
> * @cq: Used to return pointer to CQ.<br>
> * @cq_context: Used to return consumer-supplied CQ
context.<br>
> *<br>
> * All completion events returned by ibv_get_cq_event()
must<br>
> * eventually be acknowledged with ibv_ack_cq_events().<br>
> */<br>
> extern int ibv_get_cq_event(struct ibv_comp_channel *channel,<br>
>
<x-tab> </x-tab><x-tab>
</x-tab><x-tab>
</x-tab>
struct ibv_cq **cq, void <br>
> **cq_context); _______________________________________________<br>
> openib-general mailing list<br>
> openib-general@openib.org<br>
>
<a href="http://openib.org/mailman/listinfo/openib-general" eudora="autourl">
http://openib.org/mailman/listinfo/openib-general</a><br>
> <br>
> To unsubscribe, please visit <br>
>
<a href="http://openib.org/mailman/listinfo/openib-general" eudora="autourl">
http://openib.org/mailman/listinfo/openib-general</a><br>
> <br>
> <br><br>
_______________________________________________<br>
openib-general mailing list<br>
openib-general@openib.org<br>
<a href="http://openib.org/mailman/listinfo/openib-general" eudora="autourl">
http://openib.org/mailman/listinfo/openib-general</a><br><br>
To unsubscribe, please visit
<a href="http://openib.org/mailman/listinfo/openib-general" eudora="autourl">
http://openib.org/mailman/listinfo/openib-general</a></font></blockquote>
</body>
</html>