[ofw] partial review of mlx4 branch

[Hefty, Sean]

> 1. Winverbs
> I don't think I broke IBA. Please, show me where, if I missed something.
> Increasing of the version is needed to tell about new field 'Transport'.

Yes, the new field breaks the ABI.  Winverbs is designed around COM.  Any extensions must continue to support the current APIs / ABIs, so that existing applications continue to work.  That's the primary motivation behind using COM.

> It's really IBAT_EX is dependent on them, while Winverbs is dependent on
> IBAT_EX.

Winverbs can achieve the same effect by communicating directly with the kernel, like it does today.  It doesn't really matter if ibat_ex.dll changes its implementation, the user to kernel interface cannot.  Winverbs is trying to move away from ibal, adding a library dependency on it goes the opposite direction.

Note that I did NOT look at the ibat_ex.dll implementation.  I'm simply comparing what winverbs did versus the current change. 

> The latter is needed to add a seamless support to RoCE.
> Winverbs.dll is supplied as a part of OFED suit; it was dependent on kernel
> IBAT service anyway and I don't think it is too bad if it will be dependent on
> other dlls of the suit.

Winverbs already has a user interface that translates IP addresses into device addresses.  Why wouldn't you just implement ibat there, versus introducing a new API, yet another library, and requiring everyone to add dependencies on that, ibal, and complib?

> From the other side, it's not good when different user space components use
> IBAT service as they want.

Why not?  Once exposed, the user to kernel interface is usable by *any* application, which is why backwards compatibility must be maintained.  Just because we provide a library interface to the kernel doesn't preclude another developer from writing their own library that uses the same interfaces, and we have no way of knowing whether anyone has done that or not.  (I know on Linux that they have.)

> In OFED stack it is implemented inside IPoIB driver, supports only IB
> transport and is present once per machine.
> We need it to support two transports today - IB and RoCE - and more in the
> future.
> We need to deal with situation when there is no IPoIB driver loaded.

Where it's located shouldn't matter.  As long as the existing IOCTLs work, compatibility is maintained.

> 3. IBA
> What IBAs were broken ?
> How do you suggest to extend functionality while keeping IBA intact ?
> Or I misunderstood your idea ?

It's API / ABI - application programmer interface and application binary interface.  (Maybe this is just a spell check correction, but just to be clear.)  Anytime you change the structures visible to the application, the ABI has a high chance of breaking.  You can avoid breaking the ABI by:

- If the library allocates the structure, then adding new fields to the end of existing structures works.
- If you have a versioned interface, like winverbs does, then you only use the extended structure on request.
- You provide the same data through new calls.

We can see how Linux libibverbs exports rocee, then seeing if we can duplicate the same on windows.  (I'm referring to Roland's tree, not the OFED-Linux version.)

- Sean