[openfabrics-ewg] FW: Mstflint - not working on ppc64 andwhendriver is not loaded on AMD

[Tseng-Hui (Frank) Lin]

The ppc64 problem is actually in pci_64.c. Here is the patch:

============ cut here =============
diff --git a/arch/powerpc/kernel/pci_64.c b/arch/powerpc/kernel/pci_64.c
index 4c4449b..490403c 100644
--- a/arch/powerpc/kernel/pci_64.c
+++ b/arch/powerpc/kernel/pci_64.c
@@ -734,9 +734,7 @@ static struct resource *__pci_mmap_make_
 	if (hose == 0)
 		return NULL;		/* should never happen */
 
-	/* If memory, add on the PCI bridge address offset */
 	if (mmap_state == pci_mmap_mem) {
-		*offset += hose->pci_mem_offset;
 		res_bit = IORESOURCE_MEM;
 	} else {
 		io_offset = (unsigned long)hose->io_base_virt - pci_io_base;
============= end cut =============

The mmap() system call on resource0 does not work on ppc64 without this
patch. PowerMAC G5 got away with this because its hose->pci_mem_offset
was set to 0.

The fix is made on 8/21. It may be able to make it into 2.6.19. But it
certainly won't get into SLES10, SLES9-SP3, or REHL4-U4 which have
already been released. 

To cover both cases with and without the fix, my patch try to
mmap /sys/bus/pci/..../resource0 first. It it failed it tries
mmap /proc/bus/pci/.... If it failed again, we have no choice but fall
back to use PCI config space.


On Thu, 2006-09-28 at 16:59 +0300, Moshe Kazir wrote:
> Michael,
> 
> Frank found the cause to the problem in the implementation of
> arch/ppc/kernel/pci.c , 
> and asked the IBM kernel group to send a bug fix to the Linux kernel
> group.
> 
> The problem is :
> 
> 1. This bug fix will not enter SLES10 as it is closed.
> 2. It also will not enter SLES9 :-) or Redhate as4 u4 .
> 
> So we need a bug fix that will enable the use of mstflint on js21 PPC64
> + backport to old systems  .
> 
> Franks fix is based on two points (if I understand the code with no
> errors) -
> 
> 1. It opens /proc/bus/pci... And not /sys/bus/pci/...
> 2. It perform an ictl(fd, PCIIOC_MMAP_IS_MEM) ;
> 
> Frank - am I write ?
> 
> Can we enter these two small changes to the mstflint to have it working
> on the PPC64 js21 ?
> 
> Moshe 
> 
> 
> 
> 
> 
> ____________________________________________________________
> Moshe Katzir   |  +972-9971-8639 (o)   |   +972-52-860-6042  (m)
>  
> Voltaire - The Grid Backbone
>  
> www.voltaire.com
> 
>   
> 
> 
> -----Original Message-----
> From: Michael S. Tsirkin [mailto:mst at mellanox.co.il] 
> Sent: Thursday, September 28, 2006 4:41 PM
> To: Moshe Kazir
> Cc: Tseng-Hui (Frank) Lin; openfabrics-ewg at openib.org;
> openib-general at openib.org
> Subject: Re: FW: Mstflint - not working on ppc64 andwhendriver is not
> loaded on AMD
> 
> 
> Quoting r. Moshe Kazir <moshek at voltaire.com>:
> > 
> > Quoting r. Moshe Kazir <moshek at voltaire.com>:
> > > Subject: RE: FW: Mstflint - not working on ppc64 andwhendriver is 
> > > not
> > > loaded on AMD
> > > 
> > > 
> > >  # ls /sys/class/infiniband/mthca0/device/resource0
> > > /sys/class/infiniband/mthca0/device/resource0
> > 
> > OK, so can you try this please:
> > 
> > strace -f -v -o log  mstflint -d 
> > /sys/class/infiniband/mthca0/device/resource0 q
> > 
> > cat log
> > 
> > --
> > MST
> 
> 
> 
> > 30463 open("/sys/class/infiniband/mthca0/device/resource0",
> O_RDWR|O_SYNC|O_LARGEFILE) = 3
> > 30463 mmap2(NULL, 1048576, PROT_READ|PROT_WRITE, MAP_SHARED, 3, 0) =
> -1 EINVAL (Invalid argument)
> 
> So we see that mmap is failing with EINVAL.
> But why? We seem to be passing all valid parameters to it.
> 
> I'm looking at arch/ppc/kernel/pci.c at the moment.
> It seems that EINVAL is returned if __pci_mmap_make_offset
> fails, and that seems to be only looking for a valid resource size.
> 
> Are you up to finding the root cause of the problem in
> arch/ppc/kernel/pci.c?
> 
> Maybe the resource offsets are wrong? What does
> cat /sys/class/infiniband/mthca0/device/resource
> show?
> 
> Maybe there's some problem to map a full megabyte?
> Here's a test that only maps 4K. Could you strace it please?
> 
> >>>>>>>>>>>
> 
> #define _XOPEN_SOURCE 500
> #define _FILE_OFFSET_BITS 64
> 
> #include <stdio.h>
> 
> #include <unistd.h>
> 
> #include <netinet/in.h>
> #include <endian.h>
> #include <byteswap.h>
> #include <errno.h>
> #include <fcntl.h>
> #include <string.h>
> #include <stdlib.h>
> 
> #include <sys/pci.h>
> #include <sys/ioctl.h>
> 
> #include <sys/mman.h>
> #include <sys/pci.h>
> #include <sys/stat.h>
> /* #include <sys/ioctl.h>
>  * #include <sys/types.h> */
> 
> int main()
> {
>         int fd;
>         unsigned value;
>         volatile void *ptr;
>         fd = open("/proc/bus/pci/00/00.0" ,O_RDWR | O_SYNC);
> 
>         /* ioctl(fd, PCIIOC_MMAP_IS_MEM); */
>         ptr = mmap(NULL, 0x1000, PROT_READ | PROT_WRITE, MAP_SHARED, fd,
> 0xf0000);
>         memcpy(&value, (void*)(ptr + 0x14), sizeof value);
>         printf("0x%x\n");
>         return 0;
> }
> 
> 
>