[ofa-general] Re: [PATCH RFC] sharing userspace IB objects

[Michael S. Tsirkin]

> Quoting Roland Dreier <rdreier at cisco.com>:
> Subject: Re: [PATCH RFC] sharing userspace IB objects
> 
>  > This is not directly related to SRC: this is an effort
>  > to make it possible to share QPs, CQ etc across processes
>  > in the same way as they can be currently shared across threads.
>  > So assuming that we want multiple processes to post to
>  > the same QP, how can we support this?
> 
> This looks like a lot of work for an unknown gain.  Who is going to
> really use this?  ie is it worth the trouble?

It's a valid question. But let's discuss this separately.
Below are my ideas about the implementation questions that you raise.

>  > >  - Given that everything shared is in shared memory,
>  > 
>  > I think we should try and keep shared memory usage to minimum.
>  > For example, in mthca mr object just needs a key: we could
>  > keep it in non-shared memory, just pass the key around
>  > and save on sahred memory usage.
> 
> This comment made me realize there are a few more problems here.  What
> happens if I do ibv_reg_mr() in one process, pass the MR to another
> process, and then do ibv_dereg_mr() in the second process?

Generally, I think it would be nice if this could work
in the same way as with multiple threads: a single process does
destroy, the rest must not use the same object after this,
synchronisation it up to the app.

But you made me realise that we need an API for non-controlling processes to
release the userspace resources without destroying the kernel-level object.

> What about
> if someone registers a region in shared memory -- are there any
> fork/copy-on-write issues with that?

This can be done already, can't it?

> I think there are probably bugs
> in the locked_vm accounting in the kernel right now -- it doesn't take
> into account the possibility of passing context fds from one process
> to another.

Hmm, might be a good idea to fix the bugs anyway, no?

> In general what do you think the rules for destroying objects should
> be?  What if process A creates a QP, passes it to process B, and then
> process A dies?  Should the QP still be usable?

Yes, I think it should - we get this for free since file won't
be closed until both die, right?

> Should process B be
> able to destroy it?  What if process A is still alive -- should
> process B be able to destroy the QP?

I think in practice a single process will do this.
My approach generally is: let's have same rules as for multiple threads.

>  > We need to share file descriptors too. Is there a way to pass these
>  > around besides unix domain sockets?
> 
> I guess we need this to be able to re-mmap doorbell pages etc, right?
> I wonder if there's a better way around that... maybe extending the
> kernel interface so that unrelated processes can share a context, eg
> by putting contexts in a filesystem or something like that.

Hmm, I don't have principal objection, however this would mean
we'd have to change kernel-user interface again. the proposed
API extensions can mostly be done in userspace only.

And it seems to me like much more work that just let the app
use unix domain sockets, for me. What are the advantages of this approach?

Further, since there is already an existing kernel interface for this,
should we be inventing our own?

-- 
MST