[openib-general] Re: mstflint failing on sparc64

[Grant Grundler]

On Sat, Jan 08, 2005 at 10:37:41PM +0200, Michael S. Tsirkin wrote:
> We always can do the config space hack, but its much slower
> and has the disadvantage of not being atomic (two config accesses
> are needed for one memory access) and other disadvantages.

yeah - config can be *very* slow. parisc-linux port wasn't
relocating the PDC (system firmware) to RAM and each
PDC Config space access was costing 20ms (yes, *milli*seconds).
ie 32 seconds to upload tg3 firmware (1600 cfg writes). I rewrote
the PCI support to go directly to HW (PCI Host bus controller)
and we are back to microseconds again.

> OK, I'll write a small test that just maps the memory,
> to make sure there's no other bug.

I'm not sure that is necessary. I expect PCI bus resources to
be "normalized" to take the host:bus mappings into account.
PCI bus support is responsible for fixing up the advertised "resources"
to be host addresses. My point is use "resources" that show up in
/proc/iomem and do not directly access PCI BARs for this.

I previously reccomended studying the pciutils package.
But I looked up http://atrey.karlin.mff.cuni.cz/~mj/pciutils.shtml
and it's clearly only intended for PCI Config space access.

I don't know if there is some standard way to get a device's
resources other than fishing through /proc/iomem.
Ah, it's under /sys as well - /sys/bus/pci/devices/0000:30:02.0/resources.

64-bit PARISC (and ISTR alpha and sparc64) do not have 1:1 host:bus
mappings. E.g:
...
fffffffff8000000-fffffffff87fffff : PCI00 LMMIO
...
  fffffffffb050000-fffffffffb051fff : 0000:30:02.0
    fffffffffb050000-fffffffffb051fff : sym53c8xx
...
grundler at gsyprf11:~$ lspci -vvs 30:02.0
0000:30:02.0 SCSI storage controller: LSI Logic / Symbios Logic 53C896/897 (rev 01)
        Subsystem: Hewlett-Packard Company: Unknown device 10f6
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr+ Stepping- SERR+ FastB2B-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
        Latency: 128 (4250ns min, 16000ns max), Cache Line Size: 0x10 (64 bytes)
        Interrupt: pin A routed to IRQ 322
        Region 0: I/O ports at 30000 [size=256]
        Region 1: Memory at fb054000 (64-bit, non-prefetchable) [size=1K]
        Region 3: Memory at fb050000 (64-bit, non-prefetchable) [size=8K]
        Expansion ROM at fb000000 [disabled] [size=128K]
        Capabilities: <available only to root>

grundler <514>cat resource 
0x0000000000030000 0x00000000000300ff 0x0000000000000101
0xfffffffffb054000 0xfffffffffb0543ff 0x0000000000000204
0x0000000000000000 0x0000000000000000 0x0000000000000000
0xfffffffffb050000 0xfffffffffb051fff 0x0000000000000204
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 0x0000000000000000
0xfffffffffb000000 0xfffffffffb01ffff 0x0000000000007200
grundler <515>pwd
/sys/bus/pci/devices/0000:30:02.0


Note that this is a 64-bit device assigned 32-bit PCI Bus addresses.
The host address uses 64-bits.  I expect most "larger" (defined
by having more than a dozen or so PCI slots) platforms to do this
sort of thing.

hth,
grant