[ofa-general] Re: IPoIB forwarding

[Bryan Lawver]

Here's a tcpdump of the same sequence.  The TCP MSS is 8960 and it appears 
that two payloads are queued at ipoib which combines them into a single 
17920 payload with assumingly correct IP header (40) and IB header 
(4).  The application or TCP stack does not acknowledge this double packet 
ie. it does not ACK until each of the 8960 packets are resent 
individually.  Being an IB newbie, I am guessing this combining is 
allowable but may violate TCP protocol.

09:44:04.767653 IP 172.16.13.2.1751 > wopr1.5001: P 80665:89625(8960) ack 1 
win 35
09:44:04.775729 IP 172.16.13.2.1751 > wopr1.5001: . 89625:107545(17920) ack 
1 win 35
09:44:04.775751 IP wopr1.5001 > 172.16.13.2.1751: . ack 89625 win 257
09:44:04.803046 IP 172.16.13.2.1751 > wopr1.5001: . 107545:116505(8960) ack 
1 win 35
09:44:04.803069 IP wopr1.5001 > 172.16.13.2.1751: . ack 89625 win 257 
<nop,nop,sack sack 1 {107545:116505} >
09:44:04.830370 IP 172.16.13.2.1751 > wopr1.5001: P 116505:125465(8960) ack 
1 win 35
09:44:04.830392 IP wopr1.5001 > 172.16.13.2.1751: . ack 89625 win 257 
<nop,nop,sack sack 1 {107545:125465} >
09:44:04.857685 IP 172.16.13.2.1751 > wopr1.5001: . 89625:98585(8960) ack 1 
win 35
09:44:04.857712 IP wopr1.5001 > 172.16.13.2.1751: . ack 98585 win 257 
<nop,nop,sack sack 1 {107545:125465} >
09:44:05.126062 IP 172.16.13.2.1751 > wopr1.5001: P 98585:107545(8960) ack 
1 win 35
09:44:05.126086 IP wopr1.5001 > 172.16.13.2.1751: . ack 125465 win 257


At 09:14 AM 4/26/2007, Michael S. Tsirkin wrote:
> > Quoting Bryan Lawver <lawver1 at llnl.gov>:
> > Subject: Re: IPoIB forwarding
> >
> > I don't think this sequence should occur.  My IP links have MTU of 9k 
> and I
> > have set the ib0 links to the same MTU.
>
>Not sure about this: IPoIB has different encapsulation header size.
>Maybe make IPoIB MTU bigger?
>
> > This sequence of 17k size packet
> > every eighth packet seems to match my retransmit sequence as captured by
> > tcpdump on the IP end of an iperf.  The MSS in the IP header is 8960 so
> > this would seem to be one complete IP packet and the data payload from a
> > second.
> >
> >
> > Apr 26 08:30:27 wopr0 ib0: sending packet: head 0x177 length 9004
> > connection 0x340406
> > Apr 26 08:30:27 wopr0 ib0: sending packet: head 0x178 length 9004
> > connection 0x340406
> > Apr 26 08:30:27 wopr0 ib0: sending packet: head 0x179 length 9004
> > connection 0x340406
> > Apr 26 08:30:27 wopr0 ib0: sending packet: head 0x17a length 17964
> > connection 0x340406
> > Apr 26 08:30:27 wopr0 ib0: sending packet: head 0x17b length 9004
> > connection 0x340406
> > Apr 26 08:30:27 wopr0 ib0: sending packet: head 0x17c length 9004
> > connection 0x340406
> > Apr 26 08:30:27 wopr0 ib0: sending packet: head 0x17d length 9004
> > connection 0x340406
> > Apr 26 08:30:27 wopr0 ib0: sending packet: head 0x17e length 9004
> > connection 0x340406
> > Apr 26 08:30:27 wopr0 ib0: sending packet: head 0x17f length 9004
> > connection 0x340406
> > Apr 26 08:30:27 wopr0 ib0: sending packet: head 0x180 length 9004
> > connection 0x340406
> > Apr 26 08:30:27 wopr0 ib0: sending packet: head 0x181 length 9004
> > connection 0x340406
> > Apr 26 08:30:27 wopr0 ib0: sending packet: head 0x182 length 17964
> > connection 0x340406
> > Apr 26 08:30:27 wopr0 ib0: sending packet: head 0x183 length 9004
> > connection 0x340406
> > Apr 26 08:30:27 wopr0 ib0: sending packet: head 0x184 length 9004
> > connection 0x340406
> >
>
>I see you have enabled DEBUG_DATA for IPoIB CM - this will hurt performance.
>
>--
>MST