[ofa-general] [PATCH] opensm: Parallelize (Stripe) LFT sets across switches

[Hal Rosenstock]

On Thu, Jul 23, 2009 at 11:19 AM, Jason
Gunthorpe<jgunthorpe at obsidianresearch.com> wrote:
> On Thu, Jul 23, 2009 at 03:53:07PM +0300, Yevgeny Kliteynik wrote:
>> >I would be very surprised if any implementation had a significant
>> >overhead for the actual set operation compared to the packet handling
>> >path. Certainly in our products the incremental cost of a set vs
>> >processing a DR is negligible. I expect similar results from any
>> >switch. As soon as a SMP goes into the CPU for DR or other processing
>> >there is an enormous hit.
>>
>> Whether the DR packets forwarding is done in HW, FW or SW is
>> implementation dependent. I agree with you that if the same
>> entity is processing LFT block set and DR MAD forwarding, then
>> the difference between these two operations would not be very
>> big (having said that, I'm not sure it's negligible - again,
>> it's implementation dependent).
>>
>> I don't know about all the IB switches, but in InfiniScale IV
>> (and any InfiniScaleIV-based switches out there) DR packets
>> forwarding is done in HW, so its significantly faster than
>> processing LFT block by the SMA.
>
> Yes, this is sort of what worries me.. Tuning/testing for these kind
> of switches can horribly break on other switches that have low limits
> on DR SMP processing.

The proposed algorithm is less aggressive in terms of sending DR SMPs
than the current one which seems to work well enough based on field
experience. If the concern with this approach is breaking things over
the current approach, I suppose a config variable could be added for
algorithm selection with the default being the current one. I had
decided against this originally but perhaps that would make people
feel more comfortable with the proposed algorithm in the patch
submitted.

-- Hal

> Jason
>