[openib-general] MMIO reads for IPoIB w/MSI-X

[Grant Grundler]

Hi all,
In trying to understand why netperf TCP_STREAM sucks on the ia64 platform
I'm slowly collecting more data. I didn't really have any time this
week to poke at this issue.

The follow data confirms in MSI-X mode mthca is essentially doing
no MMIO reads.

ionize:~# fgrep MSI /proc/interrupts ;  pfmon -e uc_loads_retired -k --system-wide -- /opt/netperf/netperf -t TCP_STREAM -l 10 -H 10.0.0.81 -- -s 262144 -S 262144 -m 4064 ; fgrep MSI /proc/interrupts
 64:        132          0       PCI-MSI-X  ib_mthca (comp)
 65:          2          0       PCI-MSI-X  ib_mthca (async)
 66:       2203          0       PCI-MSI-X  ib_mthca (cmd)
TCP STREAM TEST to 10.0.0.81
Recv   Send    Send                          
Socket Socket  Message  Elapsed              
Size   Size    Size     Time     Throughput  
bytes  bytes   bytes    secs.    10^6bits/sec  

262142 262142   4064    10.00    1564.40   
CPU0                         680 UC_LOADS_RETIRED
CPU1                         715 UC_LOADS_RETIRED
64:     373558          0       PCI-MSI-X  ib_mthca (comp)
65:          2          0       PCI-MSI-X  ib_mthca (async)
66:       2203          0       PCI-MSI-X  ib_mthca (cmd)


hth,
grant