***SPAM*** Re: [Xen-devel] Re: [ofa-general] Fwd: pciback module not working
subbu kl
subbukl at gmail.com
Thu Feb 12 00:20:13 PST 2009
did a quick search,
I believe its MMIO, as it is
in file - http://www.cs.fsu.edu/~baker/devices/lxr/http/source/linux/drivers/infiniband/hw/mthca/mthca_main.c
mthca_QUERY_FW <http://www.cs.fsu.edu/%7Ebaker/devices/lxr/http/ident?i=mthca_QUERY_FW>()
is resulting into
mthca_QUERY_FW() which inturn will result into
mthca_cmd_post_dbell()/mthca_cmd_post_hcr() which inturn results into
__raw_writel((__force u32) cpu_to_be32(in_param >> 32), ptr
+ offs[0]);
in the file - http://www.cs.fsu.edu/~baker/devices/lxr/http/source/linux/drivers/infiniband/hw/mthca/mthca_cmd.c
OFED people should be more helpful here to comment if I have missed out
something. Roland any clue?
~subbu
On Thu, Feb 12, 2009 at 1:31 PM, Jiang, Yunhong <yunhong.jiang at intel.com>wrote:
> Can you please share more information how will the ib_mthca do QUERY_FW?
> Through config space access? Through MMIO access? I think more information
> will be helpful. The only thing seems strange to me is, from "Memory at
> fdc00000 (64-bit, non-prefetchable) [disabled] [size=1M]" , seems the MMIO
> is disabled?
>
> Thanks
> Yunhong Jiang
>
> ------------------------------
> *From:* subbu kl [mailto:subbukl at gmail.com]
> *Sent:* 2009年2月12日 15:46
>
> *To:* Jiang, Yunhong
> *Cc:* David Brown; xen-devel at lists.xensource.com;
> general at lists.openfabrics.org
> *Subject:* Re: [Xen-devel] Re: [ofa-general] Fwd: pciback module not
> working
>
> so back to square one ?
> Why QUERY_FW should fail in domU ?
>
> ~subbu
>
> On Thu, Feb 12, 2009 at 12:30 PM, Jiang, Yunhong <yunhong.jiang at intel.com>wrote:
>
>> DomU access config space through pcibackend, so that message is ok.
>>
>> ------------------------------
>> *From:* subbu kl [mailto:subbukl at gmail.com]
>> *Sent:* 2009年2月12日 14:59
>>
>> *To:* Jiang, Yunhong
>> *Cc:* David Brown; xen-devel at lists.xensource.com;
>> general at lists.openfabrics.org
>> *Subject:* Re: [Xen-devel] Re: [ofa-general] Fwd: pciback module not
>> working
>>
>> So getting PCI config space access in domU will solve the problem ? if
>> so how can I achieve that ?
>>
>> ~subbu
>>
>> On Thu, Feb 12, 2009 at 12:26 PM, Jiang, Yunhong <yunhong.jiang at intel.com
>> > wrote:
>>
>>> Sorry that seems the original mail has tried the permissive already :$
>>> How will So how will the card do the QEUREY_FW command?Through config
>>> space or through MMIO? Following information is something strange, why all
>>> the MMIO range is disabled?
>>>
>>> Memory at fdc00000 (64-bit, non-prefetchable) [disabled] [size=1M]
>>> Memory at fd000000 (64-bit, prefetchable) [disabled] [size=8M]
>>>
>>> As for the following information, I think it should be harmless since
>>> domU has no method of config spacess access method.
>>> PCI: Fatal: No PCI config space access function found
>>>
>>> Thanks
>>> Yunhong Jiang
>>>
>>> ------------------------------
>>> *From:* subbu kl [mailto:subbukl at gmail.com]
>>> *Sent:* 2009年2月12日 14:43
>>>
>>> *To:* Jiang, Yunhong
>>> *Cc:* David Brown; xen-devel at lists.xensource.com;
>>> general at lists.openfabrics.org
>>> *Subject:* Re: [Xen-devel] Re: [ofa-general] Fwd: pciback module not
>>> working
>>>
>>> oops missed it,
>>>
>>> well now I dont see that enable permissive...message. here goes the
>>> messages what I got in dom0 while booting domU
>>>
>>> tap tap-1-51712: 2 getting info
>>> pciback: vpci: 0000:0e:00.0: assign to virtual slot 0
>>> device vif1.0 entered promiscuous mode
>>> ADDRCONF(NETDEV_UP): vif1.0: link is not ready
>>> blktap: ring-ref 9, event-channel 9, protocol 1 (x86_64-abi)
>>> PCI: Enabling device 0000:0e:00.0 (0000 -> 0002)
>>> ACPI: PCI Interrupt 0000:0e:00.0[A] -> GSI 16 (level, low) -> IRQ 16
>>> PCI: Setting latency timer of device 0000:0e:00.0 to 64
>>> ACPI: PCI interrupt for device 0000:0e:00.0 disabled
>>> ADDRCONF(NETDEV_CHANGE): vif1.0: link becomes ready
>>> xenbr0: topology change detected, propagating
>>> xenbr0: port 3(vif1.0) entering forwarding state
>>>
>>> any suspicious message ?
>>> any Idea why I get that :
>>> PCI: Fatal: No PCI config space access function found
>>> rtc: IRQ 8 is not free.
>>>
>>> message in domU bootup message ?
>>>
>>> ~subbu
>>>
>>> On Thu, Feb 12, 2009 at 11:50 AM, Jiang, Yunhong <
>>> yunhong.jiang at intel.com> wrote:
>>>
>>>> So any changes in dom0's dmesg?
>>>>
>>>>
>>>> ------------------------------
>>>> *From:* subbu kl [mailto:subbukl at gmail.com]
>>>> *Sent:* 2009年2月12日 13:52
>>>> *To:* Jiang, Yunhong
>>>> *Cc:* David Brown; xen-devel at lists.xensource.com;
>>>> general at lists.openfabrics.org
>>>> *Subject:* Re: [Xen-devel] Re: [ofa-general] Fwd: pciback module not
>>>> working
>>>>
>>>> no luck !
>>>> dmesg in XEN PV guest shows :
>>>>
>>>> ib_mthca: Mellanox InfiniBand HCA driver v1.0 (April 4, 2008)
>>>> ib_mthca: Initializing 0000:00:00.0
>>>> PCI: Enabling device 0000:00:00.0 (0000 -> 0002)
>>>> PCI: Setting latency timer of device 0000:00:00.0 to 64
>>>> ib_mthca 0000:00:00.0: QUERY_FW command failed, aborting.
>>>> ib_mthca: probe of 0000:00:00.0 failed with error -11
>>>>
>>>> even after executingh the following in dom0:
>>>>
>>>> #echo -n 0000:0e:00.0 > /sys/bus/pci/drivers/pciback/permissive
>>>>
>>>> I am getting the follwing messages on the console as part of the initial
>>>> bootup messages of the guest:
>>>>
>>>> Started domain rhel52_64_3
>>>> PCI: Fatal: No PCI config space access function found
>>>> rtc: IRQ 8 is not free.
>>>> i8042.c: No controller found.
>>>>
>>>> after executing the following in dom0 :
>>>> #xm create -c rhel52_64_3
>>>>
>>>>
>>>> so, problem persisits,
>>>>
>>>> ~subbu
>>>>
>>>>
>>>> 2009/2/12 Jiang, Yunhong <yunhong.jiang at intel.com>
>>>>
>>>>> Seems it is because PCI frontend try to write some configuration
>>>>> space that PCIback has no config_field entry to support it.
>>>>> I think you can firstly try to do as dom0's dmesg suggested: "see
>>>>> permissive attribute in sysfs" (it should be "set permissive attribute...",
>>>>> I think).
>>>>>
>>>>> BTW, where you got following log? That seems suggest config space
>>>>> function not found.
>>>>>
>>>>> PCI: Fatal: No PCI config space access function found
>>>>> rtc: IRQ 8 is not free.
>>>>> i8042.c: No controller found."
>>>>>
>>>>> -- Yunhong Jiang
>>>>>
>>>>> ------------------------------
>>>>> *From:* xen-devel-bounces at lists.xensource.com [mailto:
>>>>> xen-devel-bounces at lists.xensource.com] *On Behalf Of *subbu kl
>>>>> *Sent:* 2009年2月11日 22:18
>>>>> *To:* David Brown
>>>>> *Cc:* xen-devel at lists.xensource.com; general at lists.openfabrics.org
>>>>> *Subject:* [Xen-devel] Re: [ofa-general] Fwd: pciback module not
>>>>> working
>>>>>
>>>>> I am getting the same QUERY_FW failed on RHEL5.2 with xenxen
>>>>> paravirtualized guest with pciback module.
>>>>>
>>>>> No one seems to have tried answering this question on the list, let me
>>>>> ping xen-devel and ofed people again.
>>>>>
>>>>> after executing in dom0
>>>>> echo -n 0000:0e:00.0 > /sys/bus/pci/drivers/ib_mthca/unbind
>>>>> echo -n 0000:0e:00.0 > /sys/bus/pci/drivers/pciback/new_slot
>>>>> echo -n 0000:0e:00.0 > /sys/bus/pci/drivers/pciback/bind
>>>>>
>>>>> #dmesg
>>>>> ACPI: PCI interrupt for device 0000:0e:00.0 disabled
>>>>> tap tap-1-51712: 2 getting info
>>>>> tap tap-2-51712: 2 getting info
>>>>> pciback 0000:0e:00.0: seizing device
>>>>> PCI: Enabling device 0000:0e:00.0 (0140 -> 0142)
>>>>> ACPI: PCI Interrupt 0000:0e:00.0[A] -> GSI 16 (level, low) -> IRQ 16
>>>>> ACPI: PCI interrupt for device 0000:0e:00.0 disabled
>>>>>
>>>>> #xm create -c rhel52_64_3
>>>>>
>>>>> PCI: Fatal: No PCI config space access function found
>>>>> rtc: IRQ 8 is not free.
>>>>> i8042.c: No controller found.
>>>>>
>>>>>
>>>>> GUEST dmesg:
>>>>>
>>>>> ib_mthca: Mellanox InfiniBand HCA driver v1.0 (April 4, 2008)
>>>>> ib_mthca: Initializing 0000:00:00.0
>>>>> PCI: Enabling device 0000:00:00.0 (0000 -> 0002)
>>>>> PCI: Setting latency timer of device 0000:00:00.0 to 64
>>>>> ib_mthca 0000:00:00.0: QUERY_FW command failed, aborting.
>>>>> ib_mthca: probe of 0000:00:00.0 failed with error -11
>>>>>
>>>>> in dom0:
>>>>> Feb 11 19:44:37 p128 kernel: tap tap-3-51712: 2 getting info
>>>>> Feb 11 19:44:37 p128 kernel: pciback: vpci: 0000:0e:00.0: assign to
>>>>> virtual slot 0
>>>>> Feb 11 19:44:37 p128 kernel: device vif3.0 entered promiscuous mode
>>>>> Feb 11 19:44:37 p128 kernel: ADDRCONF(NETDEV_UP): vif3.0: link is not
>>>>> ready
>>>>> Feb 11 19:44:39 p128 kernel: blktap: ring-ref 9, event-channel 9,
>>>>> protocol 1 (x86_64-abi)
>>>>> Feb 11 19:44:48 p128 kernel: pciback 0000:0e:00.0: Driver tried to
>>>>> write to a read-only configuration space field at offset 0x44, size 2. This
>>>>> may be harmless, but if you have problems with your device:
>>>>> Feb 11 19:44:48 p128 kernel: 1) see permissive attribute in sysfs
>>>>> Feb 11 19:44:48 p128 kernel: 2) report problems to the xen-devel
>>>>> mailing list along with details of your device obtained from lspci.
>>>>> Feb 11 19:44:48 p128 kernel: PCI: Enabling device 0000:0e:00.0 (0000 ->
>>>>> 0002)
>>>>> Feb 11 19:44:48 p128 kernel: ACPI: PCI Interrupt 0000:0e:00.0[A] -> GSI
>>>>> 16 (level, low) -> IRQ 16
>>>>> Feb 11 19:44:49 p128 kernel: ACPI: PCI interrupt for device
>>>>> 0000:0e:00.0 disabled
>>>>>
>>>>>
>>>>>
>>>>> some more details - [root at p128 ~]# rpm -qa | grep xen
>>>>> kernel-xen-2.6.18-92.1.22.el5
>>>>> xen-3.0.3-64.el5_2.9
>>>>> xen-libs-3.0.3-64.el5_2.9
>>>>> xen-libs-3.0.3-64.el5_2.9
>>>>>
>>>>> [root at p128 ~]# ibv_devinfo
>>>>> hca_id: mthca0
>>>>> fw_ver: 5.3.0
>>>>> node_guid: 0002:c902:0022:cd48
>>>>> sys_image_guid: 0002:c902:0022:cd4b
>>>>> vendor_id: 0x02c9
>>>>> vendor_part_id: 25218
>>>>> hw_ver: 0x20
>>>>> board_id: MT_0370130002
>>>>> phys_port_cnt: 2
>>>>> port: 1
>>>>> state: PORT_INIT (2)
>>>>> max_mtu: 2048 (4)
>>>>> active_mtu: 512 (2)
>>>>> sm_lid: 0
>>>>> port_lid: 0
>>>>> port_lmc: 0x00
>>>>>
>>>>> port: 2
>>>>> state: PORT_DOWN (1)
>>>>> max_mtu: 2048 (4)
>>>>> active_mtu: 512 (2)
>>>>> sm_lid: 0
>>>>> port_lid: 0
>>>>> port_lmc: 0x00
>>>>>
>>>>>
>>>>> any help greatly appreciated.
>>>>>
>>>>> ~subbu
>>>>>
>>>>> On Sat, Oct 18, 2008 at 4:54 AM, David Brown <dmlb2000 at gmail.com>wrote:
>>>>>
>>>>>> Okay so my question to the openfabrics guys is, why would the OFED
>>>>>> drivers fail to read the firmware?
>>>>>>
>>>>>> Any thoughts?
>>>>>>
>>>>>> Thanks,
>>>>>> - David Brown
>>>>>>
>>>>>>
>>>>>> ---------- Forwarded message ----------
>>>>>> From: David Brown <dmlb2000 at gmail.com>
>>>>>> Date: Thu, Sep 11, 2008 at 2:24 PM
>>>>>> Subject: pciback module not working
>>>>>> To: xen-users at lists.xensource.com, xen-devel at lists.xensource.com
>>>>>>
>>>>>>
>>>>>> This issue was brought up about a year and a half ago. So I'll bring
>>>>>> it up again and see if anything happens.
>>>>>>
>>>>>> I've got an infiniband network and am attempting to pass the
>>>>>> infiniband card through the host and give it to the guest.
>>>>>> I'm working with standard CentOS 5.2 on both guest and host with their
>>>>>> provided xen (3.0.3 ish). I've also attempted to install the newest
>>>>>> Xen 3.3 and use their standard host kernel and that did the same
>>>>>> thing. The guest dmesg output in the guest is similar on both
>>>>>> permissive and normal mode.
>>>>>>
>>>>>> I'm getting issues with detecting the firmware on the card for some
>>>>>> reason...
>>>>>>
>>>>>> Any help would be appreciated.
>>>>>>
>>>>>> Thanks,
>>>>>> - David Brown
>>>>>>
>>>>>> === GUEST dmesg output ===
>>>>>> ib_mthca: Mellanox InfiniBand HCA driver v1.0 (February 28, 2008)
>>>>>> ib_mthca: Initializing 0000:00:00.0
>>>>>> PCI: Enabling device 0000:00:00.0 (0000 -> 0002)
>>>>>> PCI: Setting latency timer of device 0000:00:00.0 to 64
>>>>>> ib_mthca 0000:00:00.0: QUERY_FW command failed, aborting.
>>>>>> ib_mthca: probe of 0000:00:00.0 failed with error -11
>>>>>> =======================
>>>>>>
>>>>>> === Host modprobe.conf ===
>>>>>> alias eth0 bnx2
>>>>>> alias eth1 bnx2
>>>>>> alias scsi_hostadapter cciss
>>>>>> options pciback hide=(41:00.0)
>>>>>> =====================
>>>>>>
>>>>>> === Host lspci output ===
>>>>>> # lspci -vs 41:00.0
>>>>>> 41:00.0 InfiniBand: Mellanox Technologies MT25204 [InfiniHost III Lx
>>>>>> HCA] (rev 20)
>>>>>> Subsystem: Hewlett-Packard Company Unknown device 170a
>>>>>> Flags: fast devsel, IRQ 16
>>>>>> Memory at fdc00000 (64-bit, non-prefetchable) [disabled]
>>>>>> [size=1M]
>>>>>> Memory at fd000000 (64-bit, prefetchable) [disabled] [size=8M]
>>>>>> Capabilities: [40] Power Management version 2
>>>>>> Capabilities: [48] Vital Product Data
>>>>>> Capabilities: [90] Message Signalled Interrupts: 64bit+
>>>>>> Queue=0/5 Enable-
>>>>>> Capabilities: [84] MSI-X: Enable- Mask- TabSize=32
>>>>>> Capabilities: [60] Express Endpoint IRQ 0
>>>>>> =====================
>>>>>>
>>>>>> This makes sure it get loaded first off before anything else.
>>>>>> === Host mkinitrd cmd ===
>>>>>> # mkinitrd -f --with=pciback --preload pciback
>>>>>> /boot/initrd-2.6.18-92.1.10.el5xen.img 2.6.18-92.1.10.el5xen
>>>>>> ====================
>>>>>>
>>>>>> === Host pciback dmesg ===
>>>>>> pciback 0000:41:00.0: Driver tried to write to a read-only
>>>>>> configuration space field at offset 0x44, size 2. This may be
>>>>>> harmless, but if you have problems with your device:
>>>>>> 1) see permissive attribute in sysfs
>>>>>> 2) report problems to the xen-devel mailing list along with details of
>>>>>> your device obtained from lspci.
>>>>>> PCI: Enabling device 0000:41:00.0 (0000 -> 0002)
>>>>>> ACPI: PCI Interrupt 0000:41:00.0[A] -> GSI 16 (level, low) -> IRQ 16
>>>>>> PCI: Setting latency timer of device 0000:41:00.0 to 64
>>>>>> ACPI: PCI interrupt for device 0000:41:00.0 disabled
>>>>>> ======================
>>>>>>
>>>>>> === Host pciback dmesg (after setting it permissive) ===
>>>>>> pciback 0000:41:00.0: enabling permissive mode configuration space
>>>>>> accesses!
>>>>>> pciback 0000:41:00.0: permissive mode is potentially unsafe!
>>>>>> pciback: vpci: 0000:41:00.0: assign to virtual slot 0
>>>>>> device vif1.0 entered promiscuous mode
>>>>>> ADDRCONF(NETDEV_UP): vif1.0: link is not ready
>>>>>> blkback: ring-ref 9, event-channel 28, protocol 1 (x86_64-abi)
>>>>>> PCI: Enabling device 0000:41:00.0 (0000 -> 0002)
>>>>>> ACPI: PCI Interrupt 0000:41:00.0[A] -> GSI 16 (level, low) -> IRQ 16
>>>>>> PCI: Setting latency timer of device 0000:41:00.0 to 64
>>>>>> ACPI: PCI interrupt for device 0000:41:00.0 disabled
>>>>>> =========================================
>>>>>>
>>>>>> === Guest lspci output ===
>>>>>> # lspci -v
>>>>>> 00:00.0 InfiniBand: Mellanox Technologies MT25204 [InfiniHost III Lx
>>>>>> HCA] (rev 20)
>>>>>> Subsystem: Hewlett-Packard Company Unknown device 170a
>>>>>> Flags: fast devsel, IRQ 16
>>>>>> Memory at fdc00000 (64-bit, non-prefetchable) [disabled]
>>>>>> [size=1M]
>>>>>> Memory at fd000000 (64-bit, prefetchable) [disabled] [size=8M]
>>>>>> Capabilities: [40] Power Management version 2
>>>>>> Capabilities: [48] Vital Product Data
>>>>>> Capabilities: [90] Message Signalled Interrupts: 64bit+
>>>>>> Queue=0/5 Enable-
>>>>>> Capabilities: [84] MSI-X: Enable- Mask- TabSize=32
>>>>>> Capabilities: [60] Express Endpoint IRQ 0
>>>>>> =====================
>>>>>> _______________________________________________
>>>>>> general mailing list
>>>>>> general at lists.openfabrics.org
>>>>>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>>>>>>
>>>>>> To unsubscribe, please visit
>>>>>> http://openib.org/mailman/listinfo/openib-general
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> . . . s u b b u
>>>>> "You've got to be original, because if you're like someone else, what
>>>>> do they need you for?"
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> . . . s u b b u
>>>> "You've got to be original, because if you're like someone else, what do
>>>> they need you for?"
>>>>
>>>>
>>>
>>>
>>> --
>>> . . . s u b b u
>>> "You've got to be original, because if you're like someone else, what do
>>> they need you for?"
>>>
>>>
>>
>>
>> --
>> . . . s u b b u
>> "You've got to be original, because if you're like someone else, what do
>> they need you for?"
>>
>>
>
>
> --
> . . . s u b b u
> "You've got to be original, because if you're like someone else, what do
> they need you for?"
>
>
--
. . . s u b b u
"You've got to be original, because if you're like someone else, what do
they need you for?"
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20090212/0cf12867/attachment.html>
More information about the general
mailing list