[openib-general] ip over ib throughtput

[Andras.Horvath at cern.ch]

> Does this happen even without MSI turned on?
> 
> Can you build your drivers with CONFIG_INFINIBAND_MTHCA_DEBUG=y and
> then do "dmesg -c > /dev/null && modprobe ib_mthca" and send the
> output of "dmesg"?  Don't worry about sending the whole thing to the
> list, it can't be that giant.

I'll see what I can do about the firmware. Meanwhile, here goes (without MSI
this time):

# dmesg -c > /dev/null && modprobe ib_mthca
[ some minutes pass ]
# dmesg
ib_mthca: Mellanox InfiniBand HCA driver v0.06-pre (November 8, 2004)
ib_mthca: Initializing Mellanox Technology MT23108 InfiniHost (0000:41:00.0)
GSI 38 (level, low) -> CPU 1 (0x0100) vector 58
ACPI: PCI interrupt 0000:41:00.0[A] -> GSI 38 (level, low) -> IRQ 58
ib_mthca 0000:41:00.0: Found bridge: Mellanox Technology MT23108 PCI Bridge (0000:40:01.0)
ib_mthca 0000:41:00.0: FW version 000300000002, max commands 64
ib_mthca 0000:41:00.0: FW size 6143 KB (start a7a00000, end a7ffffff)
ib_mthca 0000:41:00.0: HCA memory size 131071 KB (start a0000000, end a7ffffff)
ib_mthca 0000:41:00.0: Max QPs: 16777216, reserved QPs: 16, entry size: 256
ib_mthca 0000:41:00.0: Max CQs: 16777216, reserved CQs: 128, entry size: 64
ib_mthca 0000:41:00.0: Max EQs: 64, reserved EQs: 1, entry size: 64
ib_mthca 0000:41:00.0: reserved MPTs: 16, reserved MTTs: 16
ib_mthca 0000:41:00.0: Max PDs: 16777216, reserved PDs: 0, reserved UARs: 1
ib_mthca 0000:41:00.0: Max QP/MCG: 16777216, reserved MGMs: 0
ib_mthca 0000:41:00.0: Flags: 00370307
ib_mthca 0000:41:00.0: profile[ 0]--10/20 @ 0x        a0000000 (size 0x 4000000)
ib_mthca 0000:41:00.0: profile[ 1]-- 0/16 @ 0x        a4000000 (size 0x 1000000)
ib_mthca 0000:41:00.0: profile[ 2]-- 7/18 @ 0x        a5000000 (size 0x  800000)
ib_mthca 0000:41:00.0: profile[ 3]-- 9/17 @ 0x        a5800000 (size 0x  800000)
ib_mthca 0000:41:00.0: profile[ 4]-- 3/16 @ 0x        a6000000 (size 0x  400000)
ib_mthca 0000:41:00.0: profile[ 5]-- 4/16 @ 0x        a6400000 (size 0x  200000)
ib_mthca 0000:41:00.0: profile[ 6]--12/15 @ 0x        a6600000 (size 0x  100000)
ib_mthca 0000:41:00.0: profile[ 7]-- 8/13 @ 0x        a6700000 (size 0x   80000)
ib_mthca 0000:41:00.0: profile[ 8]--11/ 7 @ 0x        a6780000 (size 0x    1000)
ib_mthca 0000:41:00.0: profile[ 9]-- 6/ 5 @ 0x        a6781000 (size 0x     800)
ib_mthca 0000:41:00.0: HCA memory: allocated 105990 KB/124928 KB (18938 KB free)
ib_mthca 0000:41:00.0: Allocated EQ 1 with 65536 entries
ib_mthca 0000:41:00.0: Allocated EQ 2 with 128 entries
ib_mthca 0000:41:00.0: Allocated EQ 3 with 128 entries
ib_mthca 0000:41:00.0: Setting mask 00000000000343fe for eqn 2
ib_mthca 0000:41:00.0: Setting mask 0000000000000400 for eqn 3
CPI: PCI interrupt for device 0000:41:00.0 disabled
iosapic_unregister_intr(4294967295) unbalanced
Badness in iosapic_unregister_intr at arch/ia64/kernel/iosapic.c:636

Call Trace:
 [<a00000010000f280>] show_stack+0x80/0xa0
                                sp=e0000000068dfbd0 bsp=e0000000068d11a0
 [<a00000010000f2d0>] dump_stack+0x30/0x60
                                sp=e0000000068dfda0 bsp=e0000000068d1190
 [<a00000010004ac60>] iosapic_unregister_intr+0x340/0x3a0
                                sp=e0000000068dfda0 bsp=e0000000068d1138
 [<a000000100009740>] acpi_unregister_gsi+0x20/0x40
                                sp=e0000000068dfda0 bsp=e0000000068d1120
 [<a0000001002e7ae0>] acpi_pci_irq_disable+0x180/0x1c0
                                sp=e0000000068dfda0 bsp=e0000000068d10e8
 [<a0000001004898a0>] pcibios_disable_device+0x20/0x40
                                sp=e0000000068dfdb0 bsp=e0000000068d10d0
 [<a00000010028dd80>] pci_disable_device+0xc0/0xe0
                                sp=e0000000068dfdb0 bsp=e0000000068d10a8
 [<a000000200192910>] mthca_init_one+0x230/0xd40 [ib_mthca]
                                sp=e0000000068dfdc0 bsp=e0000000068d1050
 [<a00000010028ef90>] pci_device_probe_static+0xb0/0x100
                                sp=e0000000068dfe00 bsp=e0000000068d1020
 [<a00000010028f040>] __pci_device_probe+0x60/0x80
                                sp=e0000000068dfe00 bsp=e0000000068d1000
 [<a00000010028f0b0>] pci_device_probe+0x50/0xa0
                                sp=e0000000068dfe00 bsp=e0000000068d0fd8
 [<a000000100340220>] driver_probe_device+0xa0/0x140
                                sp=e0000000068dfe00 bsp=e0000000068d0fa8
 [<a000000100340580>] driver_attach+0x120/0x180
                                sp=e0000000068dfe00 bsp=e0000000068d0f78
 [<a000000100341140>] bus_add_driver+0x120/0x200
                                sp=e0000000068dfe00 bsp=e0000000068d0f48
 [<a000000100341e40>] driver_register+0xe0/0x100
                                sp=e0000000068dfe00 bsp=e0000000068d0f30
 [<a00000010028f700>] pci_register_driver+0xc0/0xe0
                                sp=e0000000068dfe30 bsp=e0000000068d0f18
 [<a000000200193450>] mthca_init+0x30/0x60 [ib_mthca]
                                sp=e0000000068dfe30 bsp=e0000000068d0f00
 [<a0000001000ccff0>] sys_init_module+0x6f0/0x700
                                sp=e0000000068dfe30 bsp=e0000000068d0e80
 [<a00000010000a160>] ia64_ret_from_syscall+0x0/0x20
                                sp=e0000000068dfe30 bsp=e0000000068d0e80
ib_mthca: probe of 0000:41:00.0 failed with error -16

Cheers,
Andras