***SPAM*** Re: [ofa-general] OpenSM and trap 128.

[Hal Rosenstock]

On Thu, Mar 26, 2009 at 8:37 AM, Nicolas Morey Chaisemartin
<nicolas.morey-chaisemartin at ext.bull.net> wrote:
> Hi,
>
> We've noticed while setting up a new cluster a problem with OpenSM.
> As usual, there are some cable problems while plugging the cluster but one of the cable was changing state over 10 000 thousands times per second (OFF/ON) and sending each time a 128 trap to OpenSM.

That's a violation of the IBTA spec by those SMAs. There's a max rate
that traps are supposed to be generated at based on the response time.
That's even if the actual event rate exceeds that which sounds like
the case here. This issue has been known for several years now but
still is unfixed :-(

> Therefore, OpenSM is constantly rerouting the whole interconnect (every 10s or so).

Even if the trap rate issue is resolved (chewing up lots of SM CPU),
it won't do anything about this issue.

> Fixing the cable will solve our problem, but I still think something should be done about this.
>
> Though OpenSM behaviour was OK, it was really difficult to find where the performances problems came from.

There should be some log messages as to the trap rate being exceeded.
Were they not present ? Which OpenSM version ?

> All our diagnostics tools (mostly using infiniband diags) were failing to see the problem.
> Infiniband diags commands fail toward the faulty port but it was hard to say if port was faulty or if it was due to high load on the SM and dropped VL15 messages.

Yes, the only thing you would observe is VL15 drops via perfquery. The
SM is the one which should be logging the trap originator which is the
way to diagnose this issue.

> I was thinking about a solution:
> When receiving a 128 trap (and it triggers a heavy sweep) we check the faulty GUID, lid or port guid.
> If last heavy sweep was triggered by the same faulty port, we wait twice last the last waiting time before forcing the new heavy sweep.
> If it's another source or another reason, we force the heavy sweep right then and set the waiting time to 0.
>
> This way, different problem will still trigger a heavy sweep asap but if only one faulty links triggers it it'll sweep less and less often as it is pretty useless.
>
> It should solve this case but there may still be a problem when more ports have the same problem...
>
> Any idea on a way to manage this?
> An ignore mask on traps? (ignore traps for 1 specific problem for x seconds if they happen to often)

I need to think about this more before commenting.

-- Hal

> Thanks
>
> Nicolas
> _______________________________________________
> general mailing list
> general at lists.openfabrics.org
> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
>