***SPAM*** Re: [Xen-devel] Re: [ofa-general] Fwd: pciback module not working
subbu kl
subbukl at gmail.com
Mon Feb 16 01:51:13 PST 2009
anyone any clue on this ?
As I am seeing the same issue with centos 5.2 HVM guest also with xen 3.4
unstable !
~subbu
On Thu, Feb 12, 2009 at 1:50 PM, subbu kl <subbukl at gmail.com> wrote:
> did a quick search,
> I believe its MMIO, as it is
>
> in file - http://www.cs.fsu.edu/~baker/devices/lxr/http/source/linux/drivers/infiniband/hw/mthca/mthca_main.c <http://www.cs.fsu.edu/%7Ebaker/devices/lxr/http/source/linux/drivers/infiniband/hw/mthca/mthca_main.c>
> mthca_QUERY_FW <http://www.cs.fsu.edu/%7Ebaker/devices/lxr/http/ident?i=mthca_QUERY_FW>() is resulting into
>
> mthca_QUERY_FW() which inturn will result into mthca_cmd_post_dbell()/mthca_cmd_post_hcr() which inturn results into
> __raw_writel((__force u32) cpu_to_be32(in_param >> 32), ptr + offs[0]);
>
>
> in the file - http://www.cs.fsu.edu/~baker/devices/lxr/http/source/linux/drivers/infiniband/hw/mthca/mthca_cmd.c <http://www.cs.fsu.edu/%7Ebaker/devices/lxr/http/source/linux/drivers/infiniband/hw/mthca/mthca_cmd.c>
>
> OFED people should be more helpful here to comment if I have missed out
> something. Roland any clue?
>
> ~subbu
>
>
> On Thu, Feb 12, 2009 at 1:31 PM, Jiang, Yunhong <yunhong.jiang at intel.com>wrote:
>
>> Can you please share more information how will the ib_mthca do QUERY_FW?
>> Through config space access? Through MMIO access? I think more information
>> will be helpful. The only thing seems strange to me is, from "Memory at
>> fdc00000 (64-bit, non-prefetchable) [disabled] [size=1M]" , seems the MMIO
>> is disabled?
>>
>> Thanks
>> Yunhong Jiang
>>
>> ------------------------------
>> *From:* subbu kl [mailto:subbukl at gmail.com]
>> *Sent:* 2009年2月12日 15:46
>>
>> *To:* Jiang, Yunhong
>> *Cc:* David Brown; xen-devel at lists.xensource.com;
>> general at lists.openfabrics.org
>> *Subject:* Re: [Xen-devel] Re: [ofa-general] Fwd: pciback module not
>> working
>>
>> so back to square one ?
>> Why QUERY_FW should fail in domU ?
>>
>> ~subbu
>>
>> On Thu, Feb 12, 2009 at 12:30 PM, Jiang, Yunhong <yunhong.jiang at intel.com
>> > wrote:
>>
>>> DomU access config space through pcibackend, so that message is ok.
>>>
>>> ------------------------------
>>> *From:* subbu kl [mailto:subbukl at gmail.com]
>>> *Sent:* 2009年2月12日 14:59
>>>
>>> *To:* Jiang, Yunhong
>>> *Cc:* David Brown; xen-devel at lists.xensource.com;
>>> general at lists.openfabrics.org
>>> *Subject:* Re: [Xen-devel] Re: [ofa-general] Fwd: pciback module not
>>> working
>>>
>>> So getting PCI config space access in domU will solve the problem ? if
>>> so how can I achieve that ?
>>>
>>> ~subbu
>>>
>>> On Thu, Feb 12, 2009 at 12:26 PM, Jiang, Yunhong <
>>> yunhong.jiang at intel.com> wrote:
>>>
>>>> Sorry that seems the original mail has tried the permissive already :$
>>>> How will So how will the card do the QEUREY_FW command?Through config
>>>> space or through MMIO? Following information is something strange, why all
>>>> the MMIO range is disabled?
>>>>
>>>> Memory at fdc00000 (64-bit, non-prefetchable) [disabled] [size=1M]
>>>> Memory at fd000000 (64-bit, prefetchable) [disabled] [size=8M]
>>>>
>>>> As for the following information, I think it should be harmless since
>>>> domU has no method of config spacess access method.
>>>> PCI: Fatal: No PCI config space access function found
>>>>
>>>> Thanks
>>>> Yunhong Jiang
>>>>
>>>> ------------------------------
>>>> *From:* subbu kl [mailto:subbukl at gmail.com]
>>>> *Sent:* 2009年2月12日 14:43
>>>>
>>>> *To:* Jiang, Yunhong
>>>> *Cc:* David Brown; xen-devel at lists.xensource.com;
>>>> general at lists.openfabrics.org
>>>> *Subject:* Re: [Xen-devel] Re: [ofa-general] Fwd: pciback module not
>>>> working
>>>>
>>>> oops missed it,
>>>>
>>>> well now I dont see that enable permissive...message. here goes the
>>>> messages what I got in dom0 while booting domU
>>>>
>>>> tap tap-1-51712: 2 getting info
>>>> pciback: vpci: 0000:0e:00.0: assign to virtual slot 0
>>>> device vif1.0 entered promiscuous mode
>>>> ADDRCONF(NETDEV_UP): vif1.0: link is not ready
>>>> blktap: ring-ref 9, event-channel 9, protocol 1 (x86_64-abi)
>>>> PCI: Enabling device 0000:0e:00.0 (0000 -> 0002)
>>>> ACPI: PCI Interrupt 0000:0e:00.0[A] -> GSI 16 (level, low) -> IRQ 16
>>>> PCI: Setting latency timer of device 0000:0e:00.0 to 64
>>>> ACPI: PCI interrupt for device 0000:0e:00.0 disabled
>>>> ADDRCONF(NETDEV_CHANGE): vif1.0: link becomes ready
>>>> xenbr0: topology change detected, propagating
>>>> xenbr0: port 3(vif1.0) entering forwarding state
>>>>
>>>> any suspicious message ?
>>>> any Idea why I get that :
>>>> PCI: Fatal: No PCI config space access function found
>>>> rtc: IRQ 8 is not free.
>>>>
>>>> message in domU bootup message ?
>>>>
>>>> ~subbu
>>>>
>>>> On Thu, Feb 12, 2009 at 11:50 AM, Jiang, Yunhong <
>>>> yunhong.jiang at intel.com> wrote:
>>>>
>>>>> So any changes in dom0's dmesg?
>>>>>
>>>>>
>>>>> ------------------------------
>>>>> *From:* subbu kl [mailto:subbukl at gmail.com]
>>>>> *Sent:* 2009年2月12日 13:52
>>>>> *To:* Jiang, Yunhong
>>>>> *Cc:* David Brown; xen-devel at lists.xensource.com;
>>>>> general at lists.openfabrics.org
>>>>> *Subject:* Re: [Xen-devel] Re: [ofa-general] Fwd: pciback module not
>>>>> working
>>>>>
>>>>> no luck !
>>>>> dmesg in XEN PV guest shows :
>>>>>
>>>>> ib_mthca: Mellanox InfiniBand HCA driver v1.0 (April 4, 2008)
>>>>> ib_mthca: Initializing 0000:00:00.0
>>>>> PCI: Enabling device 0000:00:00.0 (0000 -> 0002)
>>>>> PCI: Setting latency timer of device 0000:00:00.0 to 64
>>>>> ib_mthca 0000:00:00.0: QUERY_FW command failed, aborting.
>>>>> ib_mthca: probe of 0000:00:00.0 failed with error -11
>>>>>
>>>>> even after executingh the following in dom0:
>>>>>
>>>>> #echo -n 0000:0e:00.0 > /sys/bus/pci/drivers/pciback/permissive
>>>>>
>>>>> I am getting the follwing messages on the console as part of the
>>>>> initial bootup messages of the guest:
>>>>>
>>>>> Started domain rhel52_64_3
>>>>> PCI: Fatal: No PCI config space access function found
>>>>> rtc: IRQ 8 is not free.
>>>>> i8042.c: No controller found.
>>>>>
>>>>> after executing the following in dom0 :
>>>>> #xm create -c rhel52_64_3
>>>>>
>>>>>
>>>>> so, problem persisits,
>>>>>
>>>>> ~subbu
>>>>>
>>>>>
>>>>> 2009/2/12 Jiang, Yunhong <yunhong.jiang at intel.com>
>>>>>
>>>>>> Seems it is because PCI frontend try to write some configuration
>>>>>> space that PCIback has no config_field entry to support it.
>>>>>> I think you can firstly try to do as dom0's dmesg suggested: "see
>>>>>> permissive attribute in sysfs" (it should be "set permissive attribute...",
>>>>>> I think).
>>>>>>
>>>>>> BTW, where you got following log? That seems suggest config space
>>>>>> function not found.
>>>>>>
>>>>>> PCI: Fatal: No PCI config space access function found
>>>>>> rtc: IRQ 8 is not free.
>>>>>> i8042.c: No controller found."
>>>>>>
>>>>>> -- Yunhong Jiang
>>>>>>
>>>>>> ------------------------------
>>>>>> *From:* xen-devel-bounces at lists.xensource.com [mailto:
>>>>>> xen-devel-bounces at lists.xensource.com] *On Behalf Of *subbu kl
>>>>>> *Sent:* 2009年2月11日 22:18
>>>>>> *To:* David Brown
>>>>>> *Cc:* xen-devel at lists.xensource.com; general at lists.openfabrics.org
>>>>>> *Subject:* [Xen-devel] Re: [ofa-general] Fwd: pciback module not
>>>>>> working
>>>>>>
>>>>>> I am getting the same QUERY_FW failed on RHEL5.2 with xenxen
>>>>>> paravirtualized guest with pciback module.
>>>>>>
>>>>>> No one seems to have tried answering this question on the list, let me
>>>>>> ping xen-devel and ofed people again.
>>>>>>
>>>>>> after executing in dom0
>>>>>> echo -n 0000:0e:00.0 > /sys/bus/pci/drivers/ib_mthca/unbind
>>>>>> echo -n 0000:0e:00.0 > /sys/bus/pci/drivers/pciback/new_slot
>>>>>> echo -n 0000:0e:00.0 > /sys/bus/pci/drivers/pciback/bind
>>>>>>
>>>>>> #dmesg
>>>>>> ACPI: PCI interrupt for device 0000:0e:00.0 disabled
>>>>>> tap tap-1-51712: 2 getting info
>>>>>> tap tap-2-51712: 2 getting info
>>>>>> pciback 0000:0e:00.0: seizing device
>>>>>> PCI: Enabling device 0000:0e:00.0 (0140 -> 0142)
>>>>>> ACPI: PCI Interrupt 0000:0e:00.0[A] -> GSI 16 (level, low) -> IRQ 16
>>>>>> ACPI: PCI interrupt for device 0000:0e:00.0 disabled
>>>>>>
>>>>>> #xm create -c rhel52_64_3
>>>>>>
>>>>>> PCI: Fatal: No PCI config space access function found
>>>>>> rtc: IRQ 8 is not free.
>>>>>> i8042.c: No controller found.
>>>>>>
>>>>>>
>>>>>> GUEST dmesg:
>>>>>>
>>>>>> ib_mthca: Mellanox InfiniBand HCA driver v1.0 (April 4, 2008)
>>>>>> ib_mthca: Initializing 0000:00:00.0
>>>>>> PCI: Enabling device 0000:00:00.0 (0000 -> 0002)
>>>>>> PCI: Setting latency timer of device 0000:00:00.0 to 64
>>>>>> ib_mthca 0000:00:00.0: QUERY_FW command failed, aborting.
>>>>>> ib_mthca: probe of 0000:00:00.0 failed with error -11
>>>>>>
>>>>>> in dom0:
>>>>>> Feb 11 19:44:37 p128 kernel: tap tap-3-51712: 2 getting info
>>>>>> Feb 11 19:44:37 p128 kernel: pciback: vpci: 0000:0e:00.0: assign to
>>>>>> virtual slot 0
>>>>>> Feb 11 19:44:37 p128 kernel: device vif3.0 entered promiscuous mode
>>>>>> Feb 11 19:44:37 p128 kernel: ADDRCONF(NETDEV_UP): vif3.0: link is not
>>>>>> ready
>>>>>> Feb 11 19:44:39 p128 kernel: blktap: ring-ref 9, event-channel 9,
>>>>>> protocol 1 (x86_64-abi)
>>>>>> Feb 11 19:44:48 p128 kernel: pciback 0000:0e:00.0: Driver tried to
>>>>>> write to a read-only configuration space field at offset 0x44, size 2. This
>>>>>> may be harmless, but if you have problems with your device:
>>>>>> Feb 11 19:44:48 p128 kernel: 1) see permissive attribute in sysfs
>>>>>> Feb 11 19:44:48 p128 kernel: 2) report problems to the xen-devel
>>>>>> mailing list along with details of your device obtained from lspci.
>>>>>> Feb 11 19:44:48 p128 kernel: PCI: Enabling device 0000:0e:00.0 (0000
>>>>>> -> 0002)
>>>>>> Feb 11 19:44:48 p128 kernel: ACPI: PCI Interrupt 0000:0e:00.0[A] ->
>>>>>> GSI 16 (level, low) -> IRQ 16
>>>>>> Feb 11 19:44:49 p128 kernel: ACPI: PCI interrupt for device
>>>>>> 0000:0e:00.0 disabled
>>>>>>
>>>>>>
>>>>>>
>>>>>> some more details - [root at p128 ~]# rpm -qa | grep xen
>>>>>> kernel-xen-2.6.18-92.1.22.el5
>>>>>> xen-3.0.3-64.el5_2.9
>>>>>> xen-libs-3.0.3-64.el5_2.9
>>>>>> xen-libs-3.0.3-64.el5_2.9
>>>>>>
>>>>>> [root at p128 ~]# ibv_devinfo
>>>>>> hca_id: mthca0
>>>>>> fw_ver: 5.3.0
>>>>>> node_guid: 0002:c902:0022:cd48
>>>>>> sys_image_guid: 0002:c902:0022:cd4b
>>>>>> vendor_id: 0x02c9
>>>>>> vendor_part_id: 25218
>>>>>> hw_ver: 0x20
>>>>>> board_id: MT_0370130002
>>>>>> phys_port_cnt: 2
>>>>>> port: 1
>>>>>> state: PORT_INIT (2)
>>>>>> max_mtu: 2048 (4)
>>>>>> active_mtu: 512 (2)
>>>>>> sm_lid: 0
>>>>>> port_lid: 0
>>>>>> port_lmc: 0x00
>>>>>>
>>>>>> port: 2
>>>>>> state: PORT_DOWN (1)
>>>>>> max_mtu: 2048 (4)
>>>>>> active_mtu: 512 (2)
>>>>>> sm_lid: 0
>>>>>> port_lid: 0
>>>>>> port_lmc: 0x00
>>>>>>
>>>>>>
>>>>>> any help greatly appreciated.
>>>>>>
>>>>>> ~subbu
>>>>>>
>>>>>> On Sat, Oct 18, 2008 at 4:54 AM, David Brown <dmlb2000 at gmail.com>wrote:
>>>>>>
>>>>>>> Okay so my question to the openfabrics guys is, why would the OFED
>>>>>>> drivers fail to read the firmware?
>>>>>>>
>>>>>>> Any thoughts?
>>>>>>>
>>>>>>> Thanks,
>>>>>>> - David Brown
>>>>>>>
>>>>>>>
>>>>>>> ---------- Forwarded message ----------
>>>>>>> From: David Brown <dmlb2000 at gmail.com>
>>>>>>> Date: Thu, Sep 11, 2008 at 2:24 PM
>>>>>>> Subject: pciback module not working
>>>>>>> To: xen-users at lists.xensource.com, xen-devel at lists.xensource.com
>>>>>>>
>>>>>>>
>>>>>>> This issue was brought up about a year and a half ago. So I'll bring
>>>>>>> it up again and see if anything happens.
>>>>>>>
>>>>>>> I've got an infiniband network and am attempting to pass the
>>>>>>> infiniband card through the host and give it to the guest.
>>>>>>> I'm working with standard CentOS 5.2 on both guest and host with
>>>>>>> their
>>>>>>> provided xen (3.0.3 ish). I've also attempted to install the newest
>>>>>>> Xen 3.3 and use their standard host kernel and that did the same
>>>>>>> thing. The guest dmesg output in the guest is similar on both
>>>>>>> permissive and normal mode.
>>>>>>>
>>>>>>> I'm getting issues with detecting the firmware on the card for some
>>>>>>> reason...
>>>>>>>
>>>>>>> Any help would be appreciated.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> - David Brown
>>>>>>>
>>>>>>> === GUEST dmesg output ===
>>>>>>> ib_mthca: Mellanox InfiniBand HCA driver v1.0 (February 28, 2008)
>>>>>>> ib_mthca: Initializing 0000:00:00.0
>>>>>>> PCI: Enabling device 0000:00:00.0 (0000 -> 0002)
>>>>>>> PCI: Setting latency timer of device 0000:00:00.0 to 64
>>>>>>> ib_mthca 0000:00:00.0: QUERY_FW command failed, aborting.
>>>>>>> ib_mthca: probe of 0000:00:00.0 failed with error -11
>>>>>>> =======================
>>>>>>>
>>>>>>> === Host modprobe.conf ===
>>>>>>> alias eth0 bnx2
>>>>>>> alias eth1 bnx2
>>>>>>> alias scsi_hostadapter cciss
>>>>>>> options pciback hide=(41:00.0)
>>>>>>> =====================
>>>>>>>
>>>>>>> === Host lspci output ===
>>>>>>> # lspci -vs 41:00.0
>>>>>>> 41:00.0 InfiniBand: Mellanox Technologies MT25204 [InfiniHost III Lx
>>>>>>> HCA] (rev 20)
>>>>>>> Subsystem: Hewlett-Packard Company Unknown device 170a
>>>>>>> Flags: fast devsel, IRQ 16
>>>>>>> Memory at fdc00000 (64-bit, non-prefetchable) [disabled]
>>>>>>> [size=1M]
>>>>>>> Memory at fd000000 (64-bit, prefetchable) [disabled] [size=8M]
>>>>>>> Capabilities: [40] Power Management version 2
>>>>>>> Capabilities: [48] Vital Product Data
>>>>>>> Capabilities: [90] Message Signalled Interrupts: 64bit+
>>>>>>> Queue=0/5 Enable-
>>>>>>> Capabilities: [84] MSI-X: Enable- Mask- TabSize=32
>>>>>>> Capabilities: [60] Express Endpoint IRQ 0
>>>>>>> =====================
>>>>>>>
>>>>>>> This makes sure it get loaded first off before anything else.
>>>>>>> === Host mkinitrd cmd ===
>>>>>>> # mkinitrd -f --with=pciback --preload pciback
>>>>>>> /boot/initrd-2.6.18-92.1.10.el5xen.img 2.6.18-92.1.10.el5xen
>>>>>>> ====================
>>>>>>>
>>>>>>> === Host pciback dmesg ===
>>>>>>> pciback 0000:41:00.0: Driver tried to write to a read-only
>>>>>>> configuration space field at offset 0x44, size 2. This may be
>>>>>>> harmless, but if you have problems with your device:
>>>>>>> 1) see permissive attribute in sysfs
>>>>>>> 2) report problems to the xen-devel mailing list along with details
>>>>>>> of
>>>>>>> your device obtained from lspci.
>>>>>>> PCI: Enabling device 0000:41:00.0 (0000 -> 0002)
>>>>>>> ACPI: PCI Interrupt 0000:41:00.0[A] -> GSI 16 (level, low) -> IRQ 16
>>>>>>> PCI: Setting latency timer of device 0000:41:00.0 to 64
>>>>>>> ACPI: PCI interrupt for device 0000:41:00.0 disabled
>>>>>>> ======================
>>>>>>>
>>>>>>> === Host pciback dmesg (after setting it permissive) ===
>>>>>>> pciback 0000:41:00.0: enabling permissive mode configuration space
>>>>>>> accesses!
>>>>>>> pciback 0000:41:00.0: permissive mode is potentially unsafe!
>>>>>>> pciback: vpci: 0000:41:00.0: assign to virtual slot 0
>>>>>>> device vif1.0 entered promiscuous mode
>>>>>>> ADDRCONF(NETDEV_UP): vif1.0: link is not ready
>>>>>>> blkback: ring-ref 9, event-channel 28, protocol 1 (x86_64-abi)
>>>>>>> PCI: Enabling device 0000:41:00.0 (0000 -> 0002)
>>>>>>> ACPI: PCI Interrupt 0000:41:00.0[A] -> GSI 16 (level, low) -> IRQ 16
>>>>>>> PCI: Setting latency timer of device 0000:41:00.0 to 64
>>>>>>> ACPI: PCI interrupt for device 0000:41:00.0 disabled
>>>>>>> =========================================
>>>>>>>
>>>>>>> === Guest lspci output ===
>>>>>>> # lspci -v
>>>>>>> 00:00.0 InfiniBand: Mellanox Technologies MT25204 [InfiniHost III Lx
>>>>>>> HCA] (rev 20)
>>>>>>> Subsystem: Hewlett-Packard Company Unknown device 170a
>>>>>>> Flags: fast devsel, IRQ 16
>>>>>>> Memory at fdc00000 (64-bit, non-prefetchable) [disabled]
>>>>>>> [size=1M]
>>>>>>> Memory at fd000000 (64-bit, prefetchable) [disabled] [size=8M]
>>>>>>> Capabilities: [40] Power Management version 2
>>>>>>> Capabilities: [48] Vital Product Data
>>>>>>> Capabilities: [90] Message Signalled Interrupts: 64bit+
>>>>>>> Queue=0/5 Enable-
>>>>>>> Capabilities: [84] MSI-X: Enable- Mask- TabSize=32
>>>>>>> Capabilities: [60] Express Endpoint IRQ 0
>>>>>>> =====================
>>>>>>> _______________________________________________
>>>>>>> general mailing list
>>>>>>> general at lists.openfabrics.org
>>>>>>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>>>>>>>
>>>>>>> To unsubscribe, please visit
>>>>>>> http://openib.org/mailman/listinfo/openib-general
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> . . . s u b b u
>>>>>> "You've got to be original, because if you're like someone else, what
>>>>>> do they need you for?"
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> . . . s u b b u
>>>>> "You've got to be original, because if you're like someone else, what
>>>>> do they need you for?"
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> . . . s u b b u
>>>> "You've got to be original, because if you're like someone else, what do
>>>> they need you for?"
>>>>
>>>>
>>>
>>>
>>> --
>>> . . . s u b b u
>>> "You've got to be original, because if you're like someone else, what do
>>> they need you for?"
>>>
>>>
>>
>>
>> --
>> . . . s u b b u
>> "You've got to be original, because if you're like someone else, what do
>> they need you for?"
>>
>>
>
>
> --
> . . . s u b b u
> "You've got to be original, because if you're like someone else, what do
> they need you for?"
>
--
. . . s u b b u
"You've got to be original, because if you're like someone else, what do
they need you for?"
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20090216/e458b341/attachment.html>
More information about the general
mailing list