[openib-general] gen2 dev branch

[Sean Hefty]

On Sat, 31 Jul 2004 18:46:50 -0700
Roland Dreier <roland at topspin.com> wrote:
>     Yaron> I don't see where is the policy vs mechanism
> 
> I guess I've said enough times that I'm not comfortable with this
> design.

I think it's important that we come to an agreement regarding the best approach to take with the GSI.  And as a developer, I would rather start by focusing on the minimal requirements/features, then determine which additional features are desired and at which layer.

* We need a way to synchronize sending MADs out QP1.
* We need a method to route MADs received on QP1 to the appropriate handler.
* There should be a common RMPP module usable over any QP (for redirection).
* A useful feature would be matching responses with requests, with automatic retries.
* For debugging purposes, you may also want the snoop all sends/receives on QP1 and/or any QP.

Are any requirements missing from this?

I've always thought of the "GSI" (layer-1) as meeting only the first two requirements.  And I think that it would be optimal if layer-1 handed a received MAD to exactly one client, who would be responsible for the buffer.

I would rather see a layer-1 registration be solicited-only or unsolicited based on version/class/method.  This is more flexible than the current proposal, which is limited to trap/report.  I don't think that layer-1 should have to know the meaning of the fields.  It should act based purely on values in the fields.

Specifically, to the proposed API, I would make class/version required for unsolicited registration only.  I would replace reg_flags with a list of methods, or route based on version/class only.

QP redirection requires allocating a PD, one or two CQs, and a QP.  The size of the CQs, rearm policy, size of the QP, etc. need to be determined.  I don't see why a user-mode app can't redirect traffic to a user-mode QP.  Because of the number of options available, I think that the user needs control over them.  However, they shouldn't be required to re-implement RMPP support, if it is needed.

For the CM, I don't see that redirection requires anything beyond the existing verbs, plus a simple layer-1 interface as mentioned.  The CM can redirect to one QP, one QP per PKey, whatever, and no RMPP support is needed.  This moves the value-add upwards into the CM.

For clients requiring RMPP support, I think that there needs to be a way to use a common RMPP module above any QP.  This can be done by adding a layer-2 set of interfaces above a simple layer-1 interface.  (The actual API may extend the existing layer-1/QP APIs, but the internal architecture would have them layered.)