[openib-general] Re: [PATCH] rdma_lat-09 and results

[Grant Grundler]

On Sat, Jun 04, 2005 at 09:12:37PM +0300, Michael S. Tsirkin wrote:
> > grundler at gsyprf3$ taskset 1 ./rdma_lat -C -n 100000
> > libibverbs: Warning: no userspace device-specific driver found for uverbs0
> >         driver search path: /usr/lib/infiniband
> > 	No IB devices found
> > 	Segmentation fault

After going two more rounds, I'm still getting the same error.
At this point I admit defeat and hope that someone could just tell
me the last bit of linker magic or env variable so it Just Works.
Or maybe a "make static" target in the Makefile....*sigh*

...
> libibverbs uses dlsym(openib_driver_init) to get entry point into low level
> driver.

yup and that part seems fine. The dl is finding libmthca.so (copy
of mthca.so) but it's still not happy with something.

> 
> dlsym documentation says:
> 
> If the executable was linked with
> the flag "-rdynamic", then the global symbols in  the  executable will
> also be used to resolve references in a dynamically loaded library.
> 
> So you need to link with -rdynamic.
> Maybe you also shall list mthca.a on linker's command line,
> and pass in -u openib_driver_init, I dont remember, exactly.

It wasn't clear to me if the -rdynamic helped.
It seems like the right thing to do.

strace showed me that the dynamic linker was trying
to load "/usr/lib/libc.so.6.1" after it had loaded
"/usr/lib/infiniband/libmthca.so". But that doesn't exist.
Proper name in /lib/libc.so.6.1. My bad.  I had omitted
"/lib" from
	export LD_LIBRARY_PATH=/usr/lib:/usr/lib/infiniband

Don't need to set LD_LIBRARY_PATH. unset that and got the same behavior.
Here's the tail end of the strace output:
...
open("/sys/class/infiniband_verbs/uverbs0/ibdev", O_RDONLY) = 3
read(3, "mthca0\n", 16384)              = 7
close(3)                                = 0
open("/proc/mounts", O_RDONLY)          = 3
fstat(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
mmap(NULL, 65536, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2000000000000000
read(3, "rootfs / rootfs rw 0 0\n/dev/root"..., 1024) = 320
close(3)                                = 0
munmap(0x2000000000000000, 65536)       = 0
lstat("/sys/class/infiniband/mthca0", {st_mode=S_IFDIR|0755, st_size=0, ...}) = 0
write(2, "libibverbs: Warning: no userspac"..., 96) = 96

This doesn't make any sense:
grundler at gsyprf3:/usr/src/openib_gen2/src/userspace/perftest$ ls -l /sys/class/infiniband/mthca0/
total 0
lrwxrwxrwx  1 root root     0 Jun 21 23:30 device -> ../../../devices/pci0000:80/0000:80:01.0/0000:81:00.0
lrwxrwxrwx  1 root root     0 Jun 21 23:30 driver -> ../../../bus/pci/drivers/ib_mthca
-r--r--r--  1 root root 16384 Jun 21 23:30 fw_ver
-r--r--r--  1 root root 16384 Jun 21 23:30 hca_type
-r--r--r--  1 root root 16384 Jun 21 23:30 hw_rev
-r--r--r--  1 root root 16384 Jun 21 23:30 node_guid
-r--r--r--  1 root root 16384 Jun 21 23:30 node_type
drwxr-xr-x  4 root root     0 Jun 21 00:39 ports
-r--r--r--  1 root root 16384 Jun 21 23:30 sys_image_guid
grundler at gsyprf3:/usr/src/openib_gen2/src/userspace/perftest$ 

Whatever libibverbs might want from .../infiniband/mthca0/ it didn't even
look for it.

write(2, "/usr/lib/infiniband\n", 20)   = 20
open("/proc/cpuinfo", O_RDONLY)         = 3
fstat(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
mmap(NULL, 65536, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2000000000000000
read(3, "processor  : 0\nvendor     : Genu"..., 1024) = 508
read(3, "", 1024)                       = 0
close(3)                                = 0
munmap(0x2000000000000000, 65536)       = 0
getpid()                                = 12794
gettimeofday({1119421686, 336816}, NULL) = 0
write(2, "No IB devices found\n", 20)   = 20
exit_group(1)                           = ?
grundler at gsyprf3:/usr/src/openib_gen2/src/userspace/perftest$ 


Any other ideas?

Remember, the goal of this exercise is to track down symbols
in libibverbs described by pfmon31 Data EAR.

thanks,
grant