<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">
<br>
<div>
<div>On Sep 23, 2013, at 9:37 AM, Orion Poplawski <<a href="mailto:orion@cora.nwra.com">orion@cora.nwra.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<blockquote type="cite"><span style="font-family: Helvetica; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; display: inline !important; float: none; ">Sep
21 12:52:11 castor kernel: ib0: transmit timeout: latency 1997 msecs</span><br style="font-family: Helvetica; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; ">
<span style="font-family: Helvetica; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; display: inline !important; float: none; ">Sep
21 12:52:11 castor kernel: ib0: queue stopped 1, tx_head 2265490, tx_tail 2265362</span></blockquote>
<br>
</div>
<div>We see these periodically - even on the newer mlx4_0 devices.</div>
<div><br>
</div>
<div>If it is happening a lot and always on the same node, the HCA probably needs replacing.</div>
<div><br>
</div>
<div>How often is it happening?</div>
<br>
<div>
<div>
<div>====================================</div>
<div><br>
</div>
<div>Susan Coulter<br>
HPC-3 Network/Infrastructure<br>
505-667-8425<br>
Increase the Peace...<br>
An eye for an eye leaves the whole world blind<br>
====================================</div>
</div>
</div>
<br>
</body>
</html>