[ofa-general] Re: IPoIB forwarding
Rick Jones
rick.jones2 at hp.com
Fri Apr 27 13:32:51 PDT 2007
Bryan Lawver wrote:
> Your right about the ipoib module not combining packets (I believed you
> without checking) but I did never the less. The ipoib_start_xmit
> routine is definitely handed a "double packet" which means that the IP
> NIC driver or the kernel is combining two packets into a single super
> jumbo packet. This issue is irrespective of the IP MTU setting because
> I have set all interfaces to 9000k yet ipoib accepts and forwards this
> 17964 packet to the next IB node and onto the TCP stack where it is
> never acknowledged. This may not have come up in prior testing because
> I am using some of the fastest IP NICs which have no trouble keeping up
> with or exceeding the bandwidth of the IB side. This issue arises
> exactly every 8 packets...(ring buffer overrun??)
>
> I will be at Sonoma for the next few days as many on this list will be.
Some NICs (esp 10G) support large receive offload - they coalesce TCP segments
from the wire/fiber into larger ones they pass up the stack. Perhaps that is
happening here?
I'm going to go out a bit on a limb, cross the streams, and include netdev,
because I suspect that if a system is acting as an IP router, one doesn't want
large receive offload enabled. That may need some discussion in netdev - it may
then require some changes to default settings or some documentation
enhancements. That or I'll learn that the stack is already dealing with the
issue...
rick jones
> bryan
>
>
>
> At 11:06 AM 4/26/2007, Michael S. Tsirkin wrote:
>
>> > Quoting Bryan Lawver <lawver1 at llnl.gov>:
>> > Subject: Re: IPoIB forwarding
>> >
>> > Here's a tcpdump of the same sequence. The TCP MSS is 8960 and it
>> appears
>> > that two payloads are queued at ipoib which combines them into a single
>> > 17920 payload with assumingly correct IP header (40) and IB header
>> > (4). The application or TCP stack does not acknowledge this double
>> packet
>> > ie. it does not ACK until each of the 8960 packets are resent
>> > individually. Being an IB newbie, I am guessing this combining is
>> > allowable but may violate TCP protocol.
>>
>> IPoIB does nothing like this - it's just a network device so
>> it sends all packets out as is.
>>
>> --
>> MST
>
>
> _______________________________________________
> general mailing list
> general at lists.openfabrics.org
> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>
> To unsubscribe, please visit
> http://openib.org/mailman/listinfo/openib-general
More information about the general
mailing list