[ofw] skipping QP states during transitions

[Fab Tillier]

>> Look at the IB spec on section 10.3
>
> I was just exploring whether any hardware, separate from the existing
> software stacks, supported 'skipping' QP states -- assuming necessary
> values for the other states were also given.  In theory, hardware could
> walk through the states internally.  The motivation is to decrease the
> time to connect QPs by reducing the number of commands that need to be
> issued to the hardware.

Assuming the hardware processes commands in the order they're submitted (at least per serializes commands by resource), the HCA driver should be able to queue up the various modify commands so that they are executed by the HCA driver back-to-back.  A failure anywhere in this process should cause all subsequent commands to fail due to the QP being in the wrong state.

A single IOCTL could generate multiple back-to-back asynchronous QP modify requests.

This would eliminate the command completion processing time from the QP modify.

However, you need to have async QP modify support in the HCA driver, and ideally the HW needs to be able to process multiple commands issued against a single QP properly (Tzachi, is this something the Mellanox HCAs support?)  Note that if the HW driver cannot process multiple commands outstanding for a single resource, the driver could queue up the commands and issue the next from the DPC as soon as the first is completed.  In the current model, the DPC has to wake up the blocked thread which does whatever post processing of the command and unwinds to the caller, who then has to make the next blocking QP modify call.

-Fab