<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 TRANSITIONAL//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; CHARSET=UTF-8">
<META NAME="GENERATOR" CONTENT="GtkHTML/3.3.2">
</HEAD>
<BODY>
<!--+GtkHTML:<DATA class="ClueFlow" key="orig" value="0">--><!--+GtkHTML:<DATA class="ClueFlow" key="signature" value="0">--><!--+GtkHTML:<DATA class="ClueFlow" key="signature_name" value="uid:1150833226..18183..6@cardanus..llnl..gov">-->Hey Sasha,<BR>
<BR>
I'm working on some (unrelated to this patch) updn performance patches, when I noticed that sometimes switch ports are not used in routing. I wrote a script (will try to submit to infiniband-diags when I clean it up) that analyzes dump_lfts to see what ports are used in routing. Here's an output chunk:<BR>
<BR>
Unbalanced Switch Port Usage: MT47396 Infiniscale-III Mellanox Technologies, 0x000b8cffff004662, 40<BR>
Port 013: 12<BR>
Port 014: 12<BR>
Port 015: 12<BR>
Port 016: 12<BR>
Port 017: 12<BR>
Port 018: 12<BR>
Port 019: 12<BR>
Port 020: 12<BR>
Port 021: 12<BR>
Port 022: 0<BR>
Port 023: 11<BR>
Port 024: 11<BR>
<BR>
In the above, example, Port 022 is simply not used for routing at all on this switch. Naturally, we think this is bad. <BR>
<BR>
After some investigation, I found out that sometimes after a heavy sweep is done, some of the ports on the switches are down/not-healthy (I assume hardware racing during bringup), and thus opensm does not route through those ports. When opensm does a later heavy resweep (I assume b/c some traps are received when those ports come up), the fact that ignore_existing_lfts is not set causes opensm to still not use the "new port". They keep the same old forwarding tables from before ignore_existing_lfts is FALSE and b/c the least hops are the same.<BR>
<BR>
There are multiple ways to deal with this. I made the attached patch which solved the problem on one of our test clusters. It's pretty simple. Store all of the "bad ports" that were found during a switch configuration. During the next heavy resweep, if some of those "bad ports" are now up, I set ignore_existing_lfts to TRUE for just that switch and reconfigure the entire switch.<BR>
<BR>
Now, I will admit, that during my performance testing on this patch, performance with a few MPI tests is actually is worse. My tests are on a system with other users and I am only using 120 of 144 nodes on this cluster. So there is some potential for slowdown outside of my control.<BR>
There is some randomness that I cannot control on which specific nodes run the job, and is the lid routing layout better/worse for that specific set of nodes (unless I run on all 144 nodes, I can't be 100% sure).<BR>
<BR>
Intuitively, we think will be better as a whole. Can you think of anything that would make this patch bad for performance?<BR>
<BR>
Al<BR>
<BR>
<!--+GtkHTML:<DATA class="ClueFlow" key="signature" value="1">--><TABLE CELLSPACING="0" CELLPADDING="0" WIDTH="100%">
<TR>
<TD>
<PRE>
<!--+GtkHTML:<DATA class="ClueFlow" clear="orig">--><!--+GtkHTML:<DATA class="ClueFlow" key="signature" value="1">-->--
<!--+GtkHTML:<DATA class="ClueFlow" key="signature" value="1">-->Albert Chu
<!--+GtkHTML:<DATA class="ClueFlow" key="signature" value="1">--><A HREF="mailto:chu11@llnl.gov">chu11@llnl.gov</A>
<!--+GtkHTML:<DATA class="ClueFlow" key="signature" value="1">-->925-422-5311
<!--+GtkHTML:<DATA class="ClueFlow" key="signature" value="1">-->Computer Scientist
<!--+GtkHTML:<DATA class="ClueFlow" key="signature" value="1">-->High Performance Systems Division
<!--+GtkHTML:<DATA class="ClueFlow" key="signature" value="1">-->Lawrence Livermore National Laboratory
</PRE>
</TD>
</TR>
</TABLE>
</BODY>
</HTML>