[openib-general] Lustre over OpenIB Gen2

[Sean Hefty]

Eric Barton wrote:
> 1. How stable is the CM API and is it supported by all OpenIB affiliated
>    vendors?

The IB CM API is stable.  Changes might occur as a result of changes to the CM 
protocol itself, but that effect is not limited to just the openib API.

The RDMA CMA API is fairly stable, but could still see minor changes.  This 
would be the better connection API to use if you want to connect using IP addresses.

> 2. I'd like to scale to >= 10,000 peer nodes; 1 RC QP per peer.  Is this going
>    to get me into trouble?
> 
>    For example, I currently create a single PD and CQ for everything, however
>    the example I've seen (cmatose.c) appears to create these separately for
>    each peer.  Is that what I should be doing too?

Cmatose is just a simple example program that I use for testing.  If you're 
trying to scale out to 10,000 nodes, you'll want to limit your resources.  For 
example, I've never been able to run cmatose with 10,000 connections without 
running out of resources on my system.

Note that the IB CM does not implement a peer to peer connection model yet, so 
you would need to establish your connections using the client/server model.

> 4. Are RDMA reads still deprecated?  Which resources hit the spotlight if I
>    chose to use them?

RDMA reads are fully supported.  Not sure what lead you to think that they were 
deprecated.

> 6. Does Gen2 present substantially the same APIs as the kernel in userspace?
>    So if I wrote a userspace equivalent of my kernel driver, could I have pure
>    userspace clients talk to kernel servers?

Most of the APIs are similar.  There shouldn't be any issues talking between 
userspace clients and kernel servers.

- Sean