[ofa-general] Re: IPoIB forwarding

[Bryan Lawver]

I don't think this sequence should occur.  My IP links have MTU of 9k and I 
have set the ib0 links to the same MTU.  This sequence of 17k size packet 
every eighth packet seems to match my retransmit sequence as captured by 
tcpdump on the IP end of an iperf.  The MSS in the IP header is 8960 so 
this would seem to be one complete IP packet and the data payload from a 
second.


Apr 26 08:30:27 wopr0 ib0: sending packet: head 0x177 length 9004 
connection 0x340406
Apr 26 08:30:27 wopr0 ib0: sending packet: head 0x178 length 9004 
connection 0x340406
Apr 26 08:30:27 wopr0 ib0: sending packet: head 0x179 length 9004 
connection 0x340406
Apr 26 08:30:27 wopr0 ib0: sending packet: head 0x17a length 17964 
connection 0x340406
Apr 26 08:30:27 wopr0 ib0: sending packet: head 0x17b length 9004 
connection 0x340406
Apr 26 08:30:27 wopr0 ib0: sending packet: head 0x17c length 9004 
connection 0x340406
Apr 26 08:30:27 wopr0 ib0: sending packet: head 0x17d length 9004 
connection 0x340406
Apr 26 08:30:27 wopr0 ib0: sending packet: head 0x17e length 9004 
connection 0x340406
Apr 26 08:30:27 wopr0 ib0: sending packet: head 0x17f length 9004 
connection 0x340406
Apr 26 08:30:27 wopr0 ib0: sending packet: head 0x180 length 9004 
connection 0x340406
Apr 26 08:30:27 wopr0 ib0: sending packet: head 0x181 length 9004 
connection 0x340406
Apr 26 08:30:27 wopr0 ib0: sending packet: head 0x182 length 17964 
connection 0x340406
Apr 26 08:30:27 wopr0 ib0: sending packet: head 0x183 length 9004 
connection 0x340406
Apr 26 08:30:27 wopr0 ib0: sending packet: head 0x184 length 9004 
connection 0x340406



At 05:46 AM 4/25/2007, Michael S. Tsirkin wrote:
> > Quoting Bryan Lawver <lawver1 at llnl.gov>:
> > Subject: IPoIB forwarding
> >
> > I have a small test bed with 2 nodes with IB/OFED1.2/connected mode and a
> > third node which has IP only and is connected to one of the IB nodes.  In
> > between are DDR IB switch and 10GE IP switch.  The node with both IP 
> and IB
> > interfaces is simply a IP router in this test setup.  The IB only node has
> > a subnet route to router node and the IP only node has a subnet route to
> > the router node.
> >
> > When I launch an Iperf test from the IB (IPoIB) node to the IP node, I get
> > very good throughput with no tuning (7.5gbs).
> >
> > When I launch from IP to the IB node, I get virtually no thorughput
> > (2.5mbs).  When I dropped the window size to 8k (iperf -w8k) the 
> throughput
> > is 750mbs.
> >
> > Any suggestions, ideas?
>
>Some troubleshooting tips:
>
>Are some packets lost on the router? Checking packet counters
>might give you a clue.
>
>Do you see some errors on one of the IB nodes?
>Set debug_level=1 module parameter for ib_ipoib, and check
>dmesg output while running the test.
>
>--
>MST