[openib-general] Initial OpenIB Architecture

[Michael S. Tsirkin]

Hello!
Quoting r. Hal Rosenstock (halr at voltaire.com) "[openib-general] Initial OpenIB Architecture":
> Hi,
>  
> Here are some slides on the initial OpenIB architecture put together by some of
> the developers. This also can be found as https://openib.org/svn/gen2/trunk/
> arch/OpenIB-Arch-041221.pdf
>  
> Comments welcome.
>  
> -- Hal
>  

Hi, Hal!
Great slides!

Couple of comments (I dont mean slides need necessarily change, but maybe
mentioned in the presentation):

- Dont we want IB 1.2 compliance (slides say IB 1.1)?
- Slides say "Allocate/map userspace doorbell pages".
  - Note that you also want to unmap these. It may seem trivial,
    but it may not be so always (there may be hotswap or reset related issues).

  - Kernel code could also get slightly better protection by using doorbell
    pages. Sure, kernel is trusted, but modularity is good.

- It might be a good idea to split the "user level infiniband services"
  so separate boxes, and map it to users. For example
	- opensm is probably the only user of mad user services
	- but I think it never uses CM and SA services
  This is important e.g. since these will have different permissions.

- I also wander why is there "ib verbs" in userspace.
  E.g. should not user space cm talk just to kernel cm?
  I kind of hope the kernel/user interface will be along
  the lines of the simplicity that we currently have with umad,
  and will not expose all the rich verbs interface.

- Memory registration: "Handle memory pinning (mostly done in userspace
  with mlock() system call)". I expect you are aware that it works
  really well for full pages only ( otherwise copy on write
  begins killing you). There could be other approaches if you are
  ready to do a system call on receive, for example, do a page walk
  and copy data if the virtual address changes.
  Could be worth it mentioning that this may need looking into.

mst