[ofa-general] Re: IPoIB forwarding

[parks]

If you are using the node as a router and using the myrinet nic then 
there is something we had to turn off. It was causing panics on 
Roadrunner. It is spelled out explicitly in the Myrinet readme....
It combines packerts.

I can tell you more monday.


At 04:26 PM 4/27/2007, Bryan Lawver wrote:
>I hit the IP NIC over the head with a hammer and turned off all 
>offload features and I no longer get the super jumbo packet and I 
>have symmetric performance.  This NIC supported "ethtool -K ethx 
>tso/tx/rx/sg on/off" and I am not sure at this time which one I 
>needed to whack but all off solved the problem.
>
>Thanks for listening and re enforcing my search process.
>
>bryan
>
>At 01:32 PM 4/27/2007, Rick Jones wrote:
>>Bryan Lawver wrote:
>>>Your right about the ipoib module not combining packets (I 
>>>believed you without checking) but I did never the less.  The 
>>>ipoib_start_xmit routine is definitely handed a "double 
>>>packet"  which means that the IP NIC driver or the kernel is 
>>>combining two packets into a single super jumbo packet.  This 
>>>issue is irrespective of the IP MTU setting because I have set all 
>>>interfaces to 9000k yet  ipoib accepts and forwards this 17964 
>>>packet to the next IB node and onto the TCP stack where it is 
>>>never acknowledged.  This may not have come up in prior testing 
>>>because I am using some of the fastest IP NICs which have no 
>>>trouble keeping up with or exceeding the bandwidth of the IB 
>>>side.  This issue arises exactly every 8 packets...(ring buffer overrun??)
>>>I will be at Sonoma for the next few days as many on this list will be.
>>
>>
>>Some NICs (esp 10G) support large receive offload - they coalesce 
>>TCP segments from the wire/fiber into larger ones they pass up the 
>>stack.  Perhaps that is happening here?
>>
>>I'm going to go out a bit on a limb, cross the streams, and include 
>>netdev, because I suspect that if a system is acting as an IP 
>>router, one doesn't want large receive offload enabled.  That may 
>>need some discussion in netdev - it may then require some changes 
>>to default settings or some documentation enhancements.  That or 
>>I'll learn that the stack is already dealing with the issue...
>>
>>rick jones
>>
>>>bryan
>>>
>>>At 11:06 AM 4/26/2007, Michael S. Tsirkin wrote:
>>>
>>>> > Quoting Bryan Lawver <lawver1 at llnl.gov>:
>>>> > Subject: Re: IPoIB forwarding
>>>> >
>>>> > Here's a tcpdump of the same sequence.  The TCP MSS is 8960 
>>>> and it appears
>>>> > that two payloads are queued at ipoib which combines them into a single
>>>> > 17920 payload with assumingly correct IP header (40) and IB header
>>>> > (4).  The application or TCP stack does not acknowledge this 
>>>> double packet
>>>> > ie. it does not ACK until each of the 8960 packets are resent
>>>> > individually.  Being an IB newbie, I am guessing this combining is
>>>> > allowable but may violate TCP protocol.
>>>>
>>>>IPoIB does nothing like this - it's just a network device so
>>>>it sends all packets out as is.
>>>>
>>>>--
>>>>MST
>>>
>>>_______________________________________________
>>>general mailing list
>>>general at lists.openfabrics.org
>>>http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>>>To unsubscribe, please visit 
>>>http://openib.org/mailman/listinfo/openib-general
>
>_______________________________________________
>general mailing list
>general at lists.openfabrics.org
>http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>
>To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20070427/70593a87/attachment.html>