[ofw] partial review of mlx4 branch

[Hefty, Sean]

> ND can be functional without users able to manually transition QP states.
> From a user perspective, I have a source and destination IP address and want
> to establish a connection.  A path record is an opaque blob to me unless I
> take a hard dependency on IB.  Perhaps modifying QP states manually should be
> a privileged operation, not something any random user can do?  That's
> something that could be done without changing the ABI, mind you, by adding
> policy in the QP transition requests.

MPI and other apps rely on the ability to transition QP states themselves without running as a privileged application.  The larger the fabric, the greater the need to move towards native addresses, rather than translating from host names -> IP addresses -> GID -> path.  MPIs also bypass the IB CM completely and implement their own CM protocol using a UD QP.  There are multiple usage models that should be supported.

> QoS policy should be enforced by the kernel driver (whether that enforcement
> depends on a privileged service or not for the actual path/QoS parameters).
> An unprivileged application shouldn't be able to do whatever it wants with its
> QP.  So whether the paths are cached in the kernel or a privileged service, I
> still don't see a need for an unprivileged application being able to
> manipulate them.

It's not a matter of manipulation, it's a matter of selection.

> I guess what I want to figure out is if I need to code up handlers to publish
> path records back to user-mode.  Does WinVerbs depend on path records being
> exposed?  Does it need to?

The librdmacm permits users to specify the paths that they want to use for a connection, and libibverbs allows users to transition their QP states directly.  In a relatively short time, the librdmacm will allow users to specify native IB addresses.  (I have no timeframe on when those features will be ported to Windows, but they're there on Linux.)

- Sean