[openib-general] Re: [PATCH] rdma_lat-09 and results
Grant Grundler
iod00d at hp.com
Tue Jun 21 23:55:10 PDT 2005
On Sat, Jun 04, 2005 at 09:12:37PM +0300, Michael S. Tsirkin wrote:
> > grundler at gsyprf3$ taskset 1 ./rdma_lat -C -n 100000
> > libibverbs: Warning: no userspace device-specific driver found for uverbs0
> > driver search path: /usr/lib/infiniband
> > No IB devices found
> > Segmentation fault
After going two more rounds, I'm still getting the same error.
At this point I admit defeat and hope that someone could just tell
me the last bit of linker magic or env variable so it Just Works.
Or maybe a "make static" target in the Makefile....*sigh*
...
> libibverbs uses dlsym(openib_driver_init) to get entry point into low level
> driver.
yup and that part seems fine. The dl is finding libmthca.so (copy
of mthca.so) but it's still not happy with something.
>
> dlsym documentation says:
>
> If the executable was linked with
> the flag "-rdynamic", then the global symbols in the executable will
> also be used to resolve references in a dynamically loaded library.
>
> So you need to link with -rdynamic.
> Maybe you also shall list mthca.a on linker's command line,
> and pass in -u openib_driver_init, I dont remember, exactly.
It wasn't clear to me if the -rdynamic helped.
It seems like the right thing to do.
strace showed me that the dynamic linker was trying
to load "/usr/lib/libc.so.6.1" after it had loaded
"/usr/lib/infiniband/libmthca.so". But that doesn't exist.
Proper name in /lib/libc.so.6.1. My bad. I had omitted
"/lib" from
export LD_LIBRARY_PATH=/usr/lib:/usr/lib/infiniband
Don't need to set LD_LIBRARY_PATH. unset that and got the same behavior.
Here's the tail end of the strace output:
...
open("/sys/class/infiniband_verbs/uverbs0/ibdev", O_RDONLY) = 3
read(3, "mthca0\n", 16384) = 7
close(3) = 0
open("/proc/mounts", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
mmap(NULL, 65536, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2000000000000000
read(3, "rootfs / rootfs rw 0 0\n/dev/root"..., 1024) = 320
close(3) = 0
munmap(0x2000000000000000, 65536) = 0
lstat("/sys/class/infiniband/mthca0", {st_mode=S_IFDIR|0755, st_size=0, ...}) = 0
write(2, "libibverbs: Warning: no userspac"..., 96) = 96
This doesn't make any sense:
grundler at gsyprf3:/usr/src/openib_gen2/src/userspace/perftest$ ls -l /sys/class/infiniband/mthca0/
total 0
lrwxrwxrwx 1 root root 0 Jun 21 23:30 device -> ../../../devices/pci0000:80/0000:80:01.0/0000:81:00.0
lrwxrwxrwx 1 root root 0 Jun 21 23:30 driver -> ../../../bus/pci/drivers/ib_mthca
-r--r--r-- 1 root root 16384 Jun 21 23:30 fw_ver
-r--r--r-- 1 root root 16384 Jun 21 23:30 hca_type
-r--r--r-- 1 root root 16384 Jun 21 23:30 hw_rev
-r--r--r-- 1 root root 16384 Jun 21 23:30 node_guid
-r--r--r-- 1 root root 16384 Jun 21 23:30 node_type
drwxr-xr-x 4 root root 0 Jun 21 00:39 ports
-r--r--r-- 1 root root 16384 Jun 21 23:30 sys_image_guid
grundler at gsyprf3:/usr/src/openib_gen2/src/userspace/perftest$
Whatever libibverbs might want from .../infiniband/mthca0/ it didn't even
look for it.
write(2, "/usr/lib/infiniband\n", 20) = 20
open("/proc/cpuinfo", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
mmap(NULL, 65536, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2000000000000000
read(3, "processor : 0\nvendor : Genu"..., 1024) = 508
read(3, "", 1024) = 0
close(3) = 0
munmap(0x2000000000000000, 65536) = 0
getpid() = 12794
gettimeofday({1119421686, 336816}, NULL) = 0
write(2, "No IB devices found\n", 20) = 20
exit_group(1) = ?
grundler at gsyprf3:/usr/src/openib_gen2/src/userspace/perftest$
Any other ideas?
Remember, the goal of this exercise is to track down symbols
in libibverbs described by pfmon31 Data EAR.
thanks,
grant
More information about the general
mailing list