[ofa-general] [Bug 458] New: Heavy interrupt rates kill UDP performance

[bugzilla-daemon at lists.openfabrics.org]

https://bugs.openfabrics.org/show_bug.cgi?id=458

           Summary: Heavy interrupt rates kill UDP performance
           Product: OpenFabrics Linux
           Version: 1.1
          Platform: X86-64
        OS/Version: RHEL 4
            Status: NEW
          Severity: major
          Priority: P2
         Component: IPoIB
        AssignedTo: bugzilla at openib.org
        ReportedBy: xma at us.ibm.com


cat /proc/cpuinfo (total of 4 processors, only dispaly one here)

processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 15
model name      : Intel(R) Xeon(R) CPU            5160  @ 3.00GHz
stepping        : 6
cpu MHz         : 3000.147
cache size      : 4096 KB
physical id     : 0
siblings        : 2
core id         : 0
cpu cores       : 2
fpu             : yes
fpu_exception   : yes
cpuid level     : 10
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall lm pni monitor
ds_cpl est tm2 cx16 xtpr
bogomips        : 6006.40
clflush size    : 64
cache_alignment : 64
address sizes   : 36 bits physical, 48 bits virtual
power management:

UDP UNIDIRECTIONAL SEND TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to
10.0.0.128 (10.0.0.128) port 0 AF_INET
Socket  Message  Elapsed      Messages
Size    Size     Time         Okay Errors   Throughput
bytes   bytes    secs            #      #   10^6bits/sec

8388608   65507   60.00      588062      0    5136.18
8388608           60.00      441589           3856.87

We have tried 8MB UDP buffer and 80MB UDP buffer, that doesn't help much.
net.core.rmem_default = 8388608
net.core.wmem_default = 8388608
net.core.rmem_max = 8388608
net.core.wmem_max = 8388608

Then we tried CPU affinity for netserver to different CPU with the ib_mtcha irq
interrupt, then the UDP receiving errors are gone. 

UDP UNIDIRECTIONAL SEND TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to
10.0.0.128 (10.0.0.128) port 0 AF_INET
Socket  Message  Elapsed      Messages
Size    Size     Time         Okay Errors   Throughput
bytes   bytes    secs            #      #   10^6bits/sec

8388608   65507   60.00      591595      0    5167.07
8388608           60.00      591595           5167.07


-- 
Configure bugmail: https://bugs.openfabrics.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.