[ewg] IPoIB to 10GbE Router Configuration

Fri Jul 1 07:55:08 PDT 2011

I'm receiving the following kernel messages when traffic flows across a server configured as router using IPoIB between an IB fabric and 10GbE ethernet storage network. I can succesfully mount a storage device across the router from an Ib connected device across IPoIB, but the reponsivness suggests there is a config issue somewhere due to intermittent hangs and poor performance.

RHEl6 & OFED1.5.3.1 and the following Mellanox HCA's

09:00.0 Ethernet controller: Mellanox Technologies MT26448 [ConnectX EN 10GigE, PCIe 2.0 5GT/s] (rev b0)
0f:00.0 InfiniBand: Mellanox Technologies MT26428 [ConnectX VPI PCIe 2.0 5GT/s - IB QDR / 10GigE] (rev b0)

 I suspect its simply a config issue but if anyone has any ideas that would be greatfully appreciated.

Regards
Wayne

Jul  1 15:23:16 dlonapahls52 kernel: ------------[ cut here ]------------
Jul  1 15:23:16 dlonapahls52 kernel: WARNING: at net/core/dev.c:1640 skb_gso_segment+0x1e7/0x2c0() (Tainted: G        W  ---------------- )
Jul  1 15:23:16 dlonapahls52 kernel: Hardware name: ProLiant DL385 G7
Jul  1 15:23:16 dlonapahls52 kernel: ipoib: caps=(0x8c20, 0x0) len=1500 data_len=1388 ip_summed=1
Jul  1 15:23:16 dlonapahls52 kernel: Modules linked in: nfs lockd fscache nfs_acl auth_rpcgss sunrpc bonding rdma_ucm(U) ib_sdp(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U) ib_cm(U) ib_sa(U) ipv6 ib_uverbs(U) ib_umad(U) iw_nes(U) libcrc32c iw_cxgb3(U) cxgb3(U) ib_qib(U) dca mlx4_ib(U) mlx4_en(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_mirror dm_region_hash dm_log dm_round_robin scsi_dh_rdac dm_multipath uinput power_meter hwmon serio_raw amd64_edac_mod edac_core edac_mce_amd i2c_piix4 sg hpilo bnx2 ext4 mbcache jbd2 sr_mod cdrom sd_mod crc_t10dif ata_generic pata_acpi pata_atiixp ahci hpsa mpt2sas scsi_transport_sas raid_class radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core dm_mod [last unloaded: scsi_wait_scan]
Jul  1 15:23:16 dlonapahls52 kernel: Pid: 0, comm: swapper Tainted: G        W  ----------------  2.6.32-71.el6.x86_64 #1
Jul  1 15:23:16 dlonapahls52 kernel: Call Trace:
Jul  1 15:23:16 dlonapahls52 kernel: <IRQ>  [<ffffffff8106b857>] warn_slowpath_common+0x87/0xc0
Jul  1 15:23:16 dlonapahls52 kernel: [<ffffffff8106b946>] warn_slowpath_fmt+0x46/0x50
Jul  1 15:23:16 dlonapahls52 kernel: [<ffffffff81410197>] skb_gso_segment+0x1e7/0x2c0
Jul  1 15:23:16 dlonapahls52 kernel: [<ffffffff81410420>] dev_hard_start_xmit+0x1b0/0x370
Jul  1 15:23:16 dlonapahls52 kernel: [<ffffffff814288aa>] sch_direct_xmit+0x15a/0x1c0
Jul  1 15:23:16 dlonapahls52 kernel: [<ffffffff81413ab8>] dev_queue_xmit+0x378/0x4a0
Jul  1 15:23:16 dlonapahls52 kernel: [<ffffffff81417ddd>] neigh_connected_output+0xbd/0x100
Jul  1 15:23:16 dlonapahls52 kernel: [<ffffffff81446d77>] ip_finish_output+0x237/0x310
Jul  1 15:23:16 dlonapahls52 kernel: [<ffffffff81446f08>] ip_output+0xb8/0xc0
Jul  1 15:23:16 dlonapahls52 kernel: [<ffffffff8144242f>] ip_forward_finish+0x4f/0x60
Jul  1 15:23:16 dlonapahls52 kernel: [<ffffffff81442630>] ip_forward+0x1f0/0x420
Jul  1 15:23:16 dlonapahls52 kernel: [<ffffffff814407dd>] ip_rcv_finish+0x12d/0x440
Jul  1 15:23:16 dlonapahls52 kernel: [<ffffffff81440d65>] ip_rcv+0x275/0x350
Jul  1 15:23:16 dlonapahls52 kernel: [<ffffffff8140f2cb>] netif_receive_skb+0x38b/0x670
Jul  1 15:23:16 dlonapahls52 kernel: [<ffffffff814277d0>] ? eth_type_trans+0x40/0x140
Jul  1 15:23:16 dlonapahls52 kernel: [<ffffffffa03657e1>] mlx4_en_lro_flush_single+0x1b1/0x280 [mlx4_en]
Jul  1 15:23:16 dlonapahls52 kernel: [<ffffffffa036590c>] mlx4_en_lro_flush+0x5c/0x70 [mlx4_en]
Jul  1 15:23:16 dlonapahls52 kernel: [<ffffffffa035ebd0>] mlx4_en_process_rx_cq+0x370/0x4b0 [mlx4_en]
Jul  1 15:23:16 dlonapahls52 kernel: [<ffffffffa035ed6b>] mlx4_en_poll_rx_cq+0x5b/0xe0 [mlx4_en]
Jul  1 15:23:16 dlonapahls52 kernel: [<ffffffffa032f5e2>] ? mlx4_cq_completion+0x42/0x90 [mlx4_core]
Jul  1 15:23:16 dlonapahls52 kernel: [<ffffffff8140fe53>] net_rx_action+0x103/0x210
Jul  1 15:23:16 dlonapahls52 kernel: [<ffffffff81073bd7>] __do_softirq+0xb7/0x1e0
Jul  1 15:23:16 dlonapahls52 kernel: [<ffffffff810d8740>] ? handle_IRQ_event+0x60/0x170
Jul  1 15:23:16 dlonapahls52 kernel: [<ffffffff810142cc>] call_softirq+0x1c/0x30
Jul  1 15:23:16 dlonapahls52 kernel: [<ffffffff81015f35>] do_softirq+0x65/0xa0
Jul  1 15:23:16 dlonapahls52 kernel: [<ffffffff810739d5>] irq_exit+0x85/0x90
Jul  1 15:23:16 dlonapahls52 kernel: [<ffffffff814cf915>] do_IRQ+0x75/0xf0
Jul  1 15:23:16 dlonapahls52 kernel: [<ffffffff81013ad3>] ret_from_intr+0x0/0x11
Jul  1 15:23:16 dlonapahls52 kernel: <EOI>  [<ffffffff8103be8b>] ? native_safe_halt+0xb/0x10
Jul  1 15:23:16 dlonapahls52 kernel: [<ffffffff8101bd3d>] default_idle+0x3d/0x90
Jul  1 15:23:16 dlonapahls52 kernel: [<ffffffff81011e96>] cpu_idle+0xb6/0x110
Jul  1 15:23:16 dlonapahls52 kernel: [<ffffffff814c17c8>] start_secondary+0x1fc/0x23f
Jul  1 15:23:16 dlonapahls52 kernel: ---[ end trace 1a39d434c2f7da4b ]---

Wayne Glanfield
AHL Infrastructure
wglanfield at maninvestments.com<mailto:wglanfield at maninvestments.com>
Tel +44 20 7144 3642
Mob +44 7899 900414

[http://www.maninvestments.com/gfx/email/ahl/ahl_logo.gif]
        Man Investments Limited | Man Research Laboratory, Eagle House, Walton Well Road | Oxford OX2 6ED | England
Visit us at: www.man.com<http://www.man.com/>

Registered in England and Wales no. 2093429
Registered address: Sugar Quay, Lower Thames Street, London EC3R 6DU, United Kingdom
Man Investments Limited is authorised and regulated by the Financial Services Authority

This email has been sent by a member of the Man group (“Man”). Man’s parent company, Man Group plc, is registered in England and Wales (company number 2921462) at Sugar Quay, Lower Thames Street, London EC3R 6DU, UK.
The contents of this email are for the named addressee(s) only. It contains information which may be confidential and privileged. If you are not the intended recipient, please notify the sender immediately, destroy this email and any attachments and do not otherwise disclose or use them. Email transmission is not a secure method of communication and Man cannot accept responsibility for the completeness or accuracy of this email or any attachments. Whilst Man makes every effort to keep its network free from viruses, it does not accept responsibility for any computer virus which might be transferred by way of this email or any attachments. This email does not constitute a request, offer, recommendation or solicitation of any kind to buy, subscribe, sell or redeem any investment instruments or to perform other such transactions of any kind. Man reserves the right to monitor, record and retain all electronic and telephone communications through its network in accordance with applicable laws and regulations. --UwQe9f5k7pI3vplngP
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/ewg/attachments/20110701/25b7a9e5/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ahl_logo.gif
Type: image/gif
Size: 659 bytes
Desc: ahl_logo.gif
URL: <http://lists.openfabrics.org/pipermail/ewg/attachments/20110701/25b7a9e5/attachment.gif>