<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><br><div><div>On May 6, 2013, at 8:57 PM, Russell Dekema wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1"><div dir="ltr"><div><div><div>Susan,<br><br>Did you have to do anything 'special' (beyond what is in the FCA documentation) to get FCA to work on your cluster? Are you running FCA with Mellanox UFM, or OFA OpenSM? <br></div></div></div></div></blockquote><div><br></div>OpenSM</div><div><br><blockquote type="cite"><div dir="ltr"><div><div><div>
<br></div>I ask because we are having trouble getting FCA to work on our (UFM) cluster. We are working with Mellanox on it, but I'd be curious to hear more about your environment.<br></div></div></div></blockquote><div><br></div>There were a couple problems - the fca verbose mca options did not appear to be functional early on. (8/2012)</div><div>I was running OpenMPI with those on in an attempt to get more data on the process.</div><div>Those options seemed to introduce problems so I stopped using them. </div><div>That may be fixed now.</div><div><br></div><div>The larger problem was that it could not handle MTU mismatches.</div><div>We had changed the base MTU on the compute nodes to 4k from 2k via "set_4k_mtu=1" in modprobe , but not on the master - where the SM was running.</div><div>FCA could not handle that, so I turned on 4k MTU on the master - that fixed it.</div><div>This may also be fixed now.</div><div> </div><div>What is the problem you are experiencing? Is fca_managerd running?</div><div><br><blockquote type="cite"><div dir="ltr"><div><div><br></div>Cheers,<br>Rusty Dekema<br>
</div>CAEN High Performance Computing<br></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, May 6, 2013 at 7:09 PM, Susan Coulter <span dir="ltr"><<a href="mailto:markus@lanl.gov" target="_blank">markus@lanl.gov</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><div><br></div>Many moons ago there was a brief discussion on this list about MXM/FCA testing.<div>
I promised to send the results of my testing once I got things working.</div><div>It's been working for quite a while on a ~600 node QDR cluster and the results are pretty remarkable.</div><div>Attached are several graphs of 2 different MPI synthetics I use to test IB performance - ring and scatter.</div>
<div><br></div><div>Scatter shows pretty good performance gains with smaller message sizes - larger message sizes pretty much stink.</div><div>Ring shows significant performance increases across the board, with a couple odd results that may need more analysis/testing.</div>
<div><br></div><div><span><scatter_inc.png></span><span><scatter512.png></span><span><scatter1024.png></span><span><scatter2048.png></span><span><scatter4096.png></span><span><ring_inc.png></span><span><ring512.png></span><span><ring1024.png></span><span><ring2048.png></span><span><ring4096.png></span></div>
<div><br><div>
<div style="word-wrap:break-word"><span style="text-indent:0px;letter-spacing:normal;font-variant:normal;text-align:-webkit-auto;font-style:normal;font-weight:normal;line-height:normal;border-collapse:separate;text-transform:none;font-size:medium;white-space:normal;font-family:Helvetica;word-spacing:0px"><div style="word-wrap:break-word">
<div>====================================</div><span class="HOEnZb"><font color="#888888"><div><br></div><div>Susan Coulter<br>HPC-3 Network/Infrastructure<br><a href="tel:505-667-8425" value="+15056678425" target="_blank">505-667-8425</a><br>
Increase the Peace...<br>An eye for an eye leaves the whole world blind<br>====================================</div></font></span></div></span></div>
</div>
<br></div></div><br>_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@lists.openfabrics.org">Users@lists.openfabrics.org</a><br>
<a href="http://lists.openfabrics.org/cgi-bin/mailman/listinfo/users" target="_blank">http://lists.openfabrics.org/cgi-bin/mailman/listinfo/users</a><br>
<br></blockquote></div><br></div>
</blockquote></div><br><div>
<span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div>====================================</div><div><br></div><div>Susan Coulter<br>HPC-3 Network/Infrastructure<br>505-667-8425<br>Increase the Peace...<br>An eye for an eye leaves the whole world blind<br>====================================</div></div></span></div></span></span>
</div>
<br></body></html>