[ofa-general] Re: IPoIB forwarding

[Bryan Lawver]

Your right about the ipoib module not combining packets (I believed you 
without checking) but I did never the less.  The ipoib_start_xmit routine 
is definitely handed a "double packet"  which means that the IP NIC driver 
or the kernel is combining two packets into a single super jumbo 
packet.  This issue is irrespective of the IP MTU setting because I have 
set all interfaces to 9000k yet  ipoib accepts and forwards this 17964 
packet to the next IB node and onto the TCP stack where it is never 
acknowledged.  This may not have come up in prior testing because I am 
using some of the fastest IP NICs which have no trouble keeping up with or 
exceeding the bandwidth of the IB side.  This issue arises exactly every 8 
packets...(ring buffer overrun??)

I will be at Sonoma for the next few days as many on this list will be.

bryan



At 11:06 AM 4/26/2007, Michael S. Tsirkin wrote:
> > Quoting Bryan Lawver <lawver1 at llnl.gov>:
> > Subject: Re: IPoIB forwarding
> >
> > Here's a tcpdump of the same sequence.  The TCP MSS is 8960 and it appears
> > that two payloads are queued at ipoib which combines them into a single
> > 17920 payload with assumingly correct IP header (40) and IB header
> > (4).  The application or TCP stack does not acknowledge this double packet
> > ie. it does not ACK until each of the 8960 packets are resent
> > individually.  Being an IB newbie, I am guessing this combining is
> > allowable but may violate TCP protocol.
>
>IPoIB does nothing like this - it's just a network device so
>it sends all packets out as is.
>
>--
>MST