<html>
<body>
<font size=3><br>
If you are using the node as a router and using the myrinet nic then
there is something we had to turn off. It was causing panics on
Roadrunner. It is spelled out explicitly in the Myrinet readme....<br>
It combines packerts.<br><br>
I can tell you more monday.<br><br>
<br>
At 04:26 PM 4/27/2007, Bryan Lawver wrote:<br>
<blockquote type=cite class=cite cite="">I hit the IP NIC over the head
with a hammer and turned off all offload features and I no longer get the
super jumbo packet and I have symmetric performance. This NIC
supported "ethtool -K ethx tso/tx/rx/sg on/off" and I am not
sure at this time which one I needed to whack but all off solved the
problem.<br><br>
Thanks for listening and re enforcing my search process.<br><br>
bryan<br><br>
At 01:32 PM 4/27/2007, Rick Jones wrote:<br>
<blockquote type=cite class=cite cite="">Bryan Lawver wrote:<br>
<blockquote type=cite class=cite cite="">Your right about the ipoib
module not combining packets (I believed you without checking) but I did
never the less. The ipoib_start_xmit routine is definitely handed a
"double packet" which means that the IP NIC driver or the
kernel is combining two packets into a single super jumbo packet.
This issue is irrespective of the IP MTU setting because I have set all
interfaces to 9000k yet ipoib accepts and forwards this 17964
packet to the next IB node and onto the TCP stack where it is never
acknowledged. This may not have come up in prior testing because I
am using some of the fastest IP NICs which have no trouble keeping up
with or exceeding the bandwidth of the IB side. This issue arises
exactly every 8 packets...(ring buffer overrun??)<br>
I will be at Sonoma for the next few days as many on this list will
be.</blockquote><br><br>
Some NICs (esp 10G) support large receive offload - they coalesce TCP
segments from the wire/fiber into larger ones they pass up the
stack. Perhaps that is happening here?<br><br>
I'm going to go out a bit on a limb, cross the streams, and include
netdev, because I suspect that if a system is acting as an IP router, one
doesn't want large receive offload enabled. That may need some
discussion in netdev - it may then require some changes to default
settings or some documentation enhancements. That or I'll learn
that the stack is already dealing with the issue...<br><br>
rick jones<br><br>
<blockquote type=cite class=cite cite="">bryan<br><br>
At 11:06 AM 4/26/2007, Michael S. Tsirkin wrote:<br><br>
<blockquote type=cite class=cite cite="">> Quoting Bryan Lawver
<lawver1@llnl.gov>:<br>
> Subject: Re: IPoIB forwarding<br>
><br>
> Here's a tcpdump of the same sequence. The TCP MSS is 8960 and
it appears<br>
> that two payloads are queued at ipoib which combines them into a
single<br>
> 17920 payload with assumingly correct IP header (40) and IB
header<br>
> (4). The application or TCP stack does not acknowledge this
double packet<br>
> ie. it does not ACK until each of the 8960 packets are resent<br>
> individually. Being an IB newbie, I am guessing this combining
is<br>
> allowable but may violate TCP protocol.<br><br>
IPoIB does nothing like this - it's just a network device so<br>
it sends all packets out as is.<br><br>
--<br>
MST</blockquote><br>
_______________________________________________<br>
general mailing list<br>
general@lists.openfabrics.org<br>
<a href="http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general" eudora="autourl">
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general</a><br>
To unsubscribe, please visit
<a href="http://openib.org/mailman/listinfo/openib-general" eudora="autourl">
http://openib.org/mailman/listinfo/openib-general</a></blockquote>
</blockquote><br>
_______________________________________________<br>
general mailing list<br>
general@lists.openfabrics.org<br>
<a href="http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general" eudora="autourl">
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general</a><br><br>
To unsubscribe, please visit
<a href="http://openib.org/mailman/listinfo/openib-general" eudora="autourl">
http://openib.org/mailman/listinfo/openib-general</a></font></blockquote>
</body>
<br>
</html>