[ofw] partial review of mlx4 branch

[Fab Tillier]

Hefty, Sean wrote on Tue, 18 Oct 2011 at 09:10:47

>> I'll work on this and submit a patch at some point.  I have something
>> rudimentary prototyped but need to polish it up.  How strongly do
>> people feel about manipulating path records in user-mode? My current
>> direction has the paths entirely managed in the kernel (including
>> caching), since path records are IB-specific and don't apply to RoCE or
>> iWARP (yes, I'm still hoping some iWARP vendors join the project,
>> <sigh>).
> 
> You're trying to add a protection against access to path records that really
> doesn't matter.  As long as users can allocate and manually transition QP
> states, where path records are kept doesn't matter.

ND can be functional without users able to manually transition QP states.  >From a user perspective, I have a source and destination IP address and want to establish a connection.  A path record is an opaque blob to me unless I take a hard dependency on IB.  Perhaps modifying QP states manually should be a privileged operation, not something any random user can do?  That's something that could be done without changing the ABI, mind you, by adding policy in the QP transition requests.

In any case, it was more a matter of simplicity - IPoIB knows about the MAC to GID mapping, and through neighbor resolution done by the IP stack the route should stay up-to-date.  Having IPoIB publish the MAC to GID mappings to a user-mode service is a lot more complicated than just tracking these things in the kernel.  Since the CM is the eventual user of the path record, it was simpler to remove the round trip to user-mode for the path record.  Right now, my prototype queries the SA.  There's nothing to prevent that query logic from being replaced with, say, a socket request to some other service, whether on the same node or a centralized entity.

>  A privileged user space
> service would be a better place to cache and manage path records given
> large fabrics and the complexities of trying to handle QoS requirements.

QoS policy should be enforced by the kernel driver (whether that enforcement depends on a privileged service or not for the actual path/QoS parameters).  An unprivileged application shouldn't be able to do whatever it wants with its QP.  So whether the paths are cached in the kernel or a privileged service, I still don't see a need for an unprivileged application being able to manipulate them.

I guess what I want to figure out is if I need to code up handlers to publish path records back to user-mode.  Does WinVerbs depend on path records being exposed?  Does it need to?

-Fab