[ofa-general] Re: [PATCH RFC] sharing userspace IB objects

[Roland Dreier]

 > This is not directly related to SRC: this is an effort
 > to make it possible to share QPs, CQ etc across processes
 > in the same way as they can be currently shared across threads.
 > So assuming that we want multiple processes to post to
 > the same QP, how can we support this?

This looks like a lot of work for an unknown gain.  Who is going to
really use this?  ie is it worth the trouble?

 > >  - Given that everything shared is in shared memory,
 > 
 > I think we should try and keep shared memory usage to minimum.
 > For example, in mthca mr object just needs a key: we could
 > keep it in non-shared memory, just pass the key around
 > and save on sahred memory usage.

This comment made me realize there are a few more problems here.  What
happens if I do ibv_reg_mr() in one process, pass the MR to another
process, and then do ibv_dereg_mr() in the second process?  What about
if someone registers a region in shared memory -- are there any
fork/copy-on-write issues with that?  I think there are probably bugs
in the locked_vm accounting in the kernel right now -- it doesn't take
into account the possibility of passing context fds from one process
to another.

In general what do you think the rules for destroying objects should
be?  What if process A creates a QP, passes it to process B, and then
process A dies?  Should the QP still be usable?  Should process B be
able to destroy it?  What if process A is still alive -- should
process B be able to destroy the QP?

 > We need to share file descriptors too. Is there a way to pass these
 > around besides unix domain sockets?

I guess we need this to be able to re-mmap doorbell pages etc, right?
I wonder if there's a better way around that... maybe extending the
kernel interface so that unrelated processes can share a context, eg
by putting contexts in a filesystem or something like that.

 > But are you sure we want to break API for all users just to add
 > a new capability for a minority that wants shared memory support?

Yes, you're right... better to be backward compatible and have a new
API for shared stuff.

 - R.