[ofa-general] IPoIB Transmit Timeouts

[Charles A. Taylor]

We upgraded our file servers to OFED 1.4.1 last Thursday and have since
been hit with a daily ration of the following across all eight of our
servers...

Aug 17 09:46:59 hpcio8 kernel: NETDEV WATCHDOG: ib1: transmit timed out
Aug 17 09:46:59 hpcio8 kernel: ib1: transmit timeout: latency 347449
msecs
Aug 17 09:46:59 hpcio8 kernel: ib1: queue stopped 1, tx_head 868165770,
tx_tail 868165647

The difference between the head/tail is always 123.   The send queue
size is 128 according to...

cat /sys/module/ib_ipoib/parameters/send_queue_size 
128

>From the post below, others seem to have encountered this but we have
not seen any patches or work-arounds.   Has anyone solved this problem?

They were very stable under OFED 1.2.   We are running the
Lustre-patched kernel but we did that under OFED 1.2 + lustre 1.6.4.2 as
well and I'm pretty sure they don't touch the IB modules.

Relevant information:
=====================
CentOS 5.3
Lustre 1.8.0.1
2.6.18-128.1.6.el5_lustre.1.8.0.1smp
X86_64 (Opteron 275s)

hca_id: mthca0
        fw_ver:                         4.8.200
        node_guid:                      0005:ad00:0004:668c
        sys_image_guid:                 0002:c900:0100:d050
        vendor_id:                      0x02c9
        vendor_part_id:                 25208
        hw_ver:                         0xA0
        board_id:                       MT_00A0000001
        phys_port_cnt:                  2
                port:   1
                        state:                  active (4)
                        max_mtu:                2048 (4)
                        active_mtu:             2048 (4)
                        sm_lid:                 1
                        port_lid:               49
                        port_lmc:               0x00

                port:   2
                        state:                  active (4)
                        max_mtu:                2048 (4)
                        active_mtu:             2048 (4)
                        sm_lid:                 1
                        port_lid:               98
                        port_lmc:               0x00


Charlie Taylor
UF HPC Center

> On Wed, Jul 29, 2009 at 2:14 PM, Pradeep Satyanarayana <
> prade... at linux.vnet.ibm.com> wrote:
> 
> > Hal Rosenstock wrote:
> > > Hi,
> > >
> > > I'm seeing the following messages from IPoIB:
> > > ib0: post_send failed
> > > ib0: post_send failed
> > > ib0: post_send failed
> > > ib0: post_send failed
> > > ib0: post_send failed
> > > ib0: post_send failed
> > > NETDEV WATCHDOG: ib0: transmit timed out
> > > ib0: transmit timeout: latency 1374 msecs
> > > ib0: queue stopped 1, tx_head 140245691, tx_tail 140245565
> > >
> > > What are the possible (and most likely) causes of post_send failures ? I
> > > went through the code for all the errors (some at the driver level) but
> > > none popped out at me.
> > >
> >
> > Is it possible that the receiver is overwhelmed and hence the tx_ring is
> > full?
>