[openib-general] Initial OpenIB Architecture

[Hal Rosenstock]

On Tue, 2005-01-11 at 10:58, Michael S. Tsirkin wrote:
> Great slides!

Thanks. The credit for this goes to all the contributors/developers.

> Couple of comments (I dont mean slides need necessarily change, but maybe
> mentioned in the presentation):
> 
> - Dont we want IB 1.2 compliance (slides say IB 1.1)?

We are _initially_ targeting IBA 1.1 with selected 1.2 compliance right
now. DDR is one thing. Client reregister and SRQ support are some
others. There are a few others which escape me right now but a more
complete list can be put together. These are on a need basis right now.
More 1.2 compliance will come later. Are there some 1.2 specifics which
you think are needed sooner ?

> - Slides say "Allocate/map userspace doorbell pages".
>   - Note that you also want to unmap these. It may seem trivial,
>     but it may not be so always (there may be hotswap or reset related issues).
>   - Kernel code could also get slightly better protection by using doorbell
>     pages. Sure, kernel is trusted, but modularity is good.

> - It might be a good idea to split the "user level infiniband services"
>   so separate boxes, and map it to users. 

Not sure what you mean by this. The user IB services are available to
any application. 

> For example
> 	- opensm is probably the only user of mad user services

It's not. The diagnostic tools are also consumers of user MAD services
as are any user space SA clients associated with applications or CM
clients in user space.

> 	- but I think it never uses CM and SA services

OpenSM also contains SA (but not SA client which is intented for those
applications needing things like PathRecords, etc.). SM/SA does not need
CM. Am I misunderstanding ?

>   This is important e.g. since these will have different permissions.

> - I also wander why is there "ib verbs" in userspace.
>   E.g. should not user space cm talk just to kernel cm?

I think that is the intention although this is TBD.

>   I kind of hope the kernel/user interface will be along
>   the lines of the simplicity that we currently have with umad,
>   and will not expose all the rich verbs interface.

This is coming out of Roland and Libor's user space work. I think it
needs more richness than umad to allow for RDMA, atomics, SRQs, etc.

> - Memory registration: "Handle memory pinning (mostly done in userspace
>   with mlock() system call)". I expect you are aware that it works
>   really well for full pages only ( otherwise copy on write
>   begins killing you). There could be other approaches if you are
>   ready to do a system call on receive, for example, do a page walk
>   and copy data if the virtual address changes.
>   Could be worth it mentioning that this may need looking into.

Thanks for the feedback. Please continue to monitor the list and discuss
issues with the evolving APIs, design, and implementation.

-- Hal