[ofa-general] Re: [PATCH RFC] sharing userspace IB objects

[Roland Dreier]

 > Generally, I think it would be nice if this could work
 > in the same way as with multiple threads: a single process does
 > destroy, the rest must not use the same object after this,
 > synchronisation it up to the app.
 > 
 > But you made me realise that we need an API for non-controlling processes to
 > release the userspace resources without destroying the kernel-level object.

What is a non-controlling process?  To the uverbs code in the kernel
there is only one file structure that happens to be shared by multiple
processes.  But they are all equal.

 > > I think there are probably bugs
 > > in the locked_vm accounting in the kernel right now -- it doesn't take
 > > into account the possibility of passing context fds from one process
 > > to another.
 > 
 > Hmm, might be a good idea to fix the bugs anyway, no?

Yes, I guess we need to take a reference on the mm structure in
ib_umem_get() and only drop it after we free the umem.

 > > Should process B be
 > > able to destroy it?  What if process A is still alive -- should
 > > process B be able to destroy the QP?
 > 
 > I think in practice a single process will do this.
 > My approach generally is: let's have same rules as for multiple threads.

I don't think it's quite as simple as saying that it's just like
multiple threads.  Creating/destroying QPs from a PD shared by
multiple processes opens lots of problems.  Let's take the mthca case:
the userspace driver needs to have a table of QPN -> QP struct so that
it can look up which QP a completion belongs to.

This means that if process A and process B share QP X, then X has to
be in the QP table of both processes.  OK, that's fine, when QP X gets
passed from A to B, then B can put it in the table.  But what happens
when B destroys QP X?  How does process A know to take X out of its table?
What if process A has died in the meantime?

Or what if process A and process B share CQ 1, and process B creates
QP Y in a non-shared PD but attaches it to CQ 1?  What happens when
process A polls a completion for QP Y from CQ 1?

 > > I guess we need this to be able to re-mmap doorbell pages etc, right?
 > > I wonder if there's a better way around that... maybe extending the
 > > kernel interface so that unrelated processes can share a context, eg
 > > by putting contexts in a filesystem or something like that.
 > 
 > Hmm, I don't have principal objection, however this would mean
 > we'd have to change kernel-user interface again. the proposed
 > API extensions can mostly be done in userspace only.
 > 
 > And it seems to me like much more work that just let the app
 > use unix domain sockets, for me. What are the advantages of this approach?
 > 
 > Further, since there is already an existing kernel interface for this,
 > should we be inventing our own?

The advantage is that sharing objects in a filesystem by doing open(),
protected by permissions etc. is much more familiar than passing fds
through sockets.  I'm not sure it makes sense but shared memory + unix
domain socket fd passing is not a very natural way for most people to
program.

 - R.