[ofa-general] Re: [PATCH RFC] sharing userspace IB objects

[Ganesh Sadasivan]

One advantage of having shared objects is to be able to preserve IB
connections across process restarts. If the traffic is not very high
and the buffers are in shared memory (which I think should be), then
it can save connection setup and message recovery time.

Shouldn't the protocol to create and destroy and pass the various
IB objects around be decided by the specific application rather than
the library trying to solve this problem?

Thanks
Ganesh

On 6/26/07, Roland Dreier <rdreier at cisco.com> wrote:
>
> > This is not directly related to SRC: this is an effort
> > to make it possible to share QPs, CQ etc across processes
> > in the same way as they can be currently shared across threads.
> > So assuming that we want multiple processes to post to
> > the same QP, how can we support this?
>
> This looks like a lot of work for an unknown gain.  Who is going to
> really use this?  ie is it worth the trouble?
>
> > >  - Given that everything shared is in shared memory,
> >
> > I think we should try and keep shared memory usage to minimum.
> > For example, in mthca mr object just needs a key: we could
> > keep it in non-shared memory, just pass the key around
> > and save on sahred memory usage.
>
> This comment made me realize there are a few more problems here.  What
> happens if I do ibv_reg_mr() in one process, pass the MR to another
> process, and then do ibv_dereg_mr() in the second process?  What about
> if someone registers a region in shared memory -- are there any
> fork/copy-on-write issues with that?  I think there are probably bugs
> in the locked_vm accounting in the kernel right now -- it doesn't take
> into account the possibility of passing context fds from one process
> to another.
>
> In general what do you think the rules for destroying objects should
> be?  What if process A creates a QP, passes it to process B, and then
> process A dies?  Should the QP still be usable?  Should process B be
> able to destroy it?  What if process A is still alive -- should
> process B be able to destroy the QP?
>
> > We need to share file descriptors too. Is there a way to pass these
> > around besides unix domain sockets?
>
> I guess we need this to be able to re-mmap doorbell pages etc, right?
> I wonder if there's a better way around that... maybe extending the
> kernel interface so that unrelated processes can share a context, eg
> by putting contexts in a filesystem or something like that.
>
> > But are you sure we want to break API for all users just to add
> > a new capability for a minority that wants shared memory support?
>
> Yes, you're right... better to be backward compatible and have a new
> API for shared stuff.
>
> - R.
> _______________________________________________
> general mailing list
> general at lists.openfabrics.org
> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>
> To unsubscribe, please visit
> http://openib.org/mailman/listinfo/openib-general
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20070627/e8537b42/attachment.html>