[ofa-general] using SDP for block device traffic: several problems

[Amir Vadai]

Lars Hi,

I opened a bug in our bugzilla (https://bugs.openfabrics.org/show_bug.cgi?id=1672).

I couldn't reproduce it on my setup: SLES 10SP2, stock kernel, same ofed git version.
will try now to install 2.6.27 kernel and check again.

BTW, what type of servers do you use? Are they low/high end server?

- Amir


On 07/07/2009 08:13 PM, Lars Ellenberg wrote:
> this is on debian lenny,
> userland ofed from http://pkg-ofed.alioth.debian.org/apt/ofed/
> 
> kernel git://git.openfabrics.org/ofed_1_4/linux-2.6.git
> merged with upstream stable
> git://git4.kernel.org/pub/scm/linux/kernel/git/stable/linux-2.6.27.y.git
> 
> both as of today:
>   ofed_kernel            08acda8 sdp: Fix memory leak in bzcopy
>   linux-v2.6.27.y/master 49cbf40 Linux 2.6.27.26
> 
> 
> kernel config: very many "kernel debugging" things enabled.
> if you want me to try a certain .config, or anything,
> this can be arranged.
> 
> two very ugly perl scripts attached,
> one tcp server,
> one tcp client,
> adapted from the perlipc man page.
> 
> client connects,
> sends a package,
> receives a package
> in an endless loop.
> 
> package format:
> 4 byte magic, 2 byte ignored, 2 byte payload length
> indicated length of payload, all same bytes,
> but the trailing 4 byte, which again is a magic number.
> 
> with ethernet, or IPoIB: runs endless.
> 
> with LD_PRELOAD=libsdp.so runs for very few iterations,
> and errors out on one of the sanity checks.
> 
> 
> sample output:
> root at kugel:/home/lars/DRBD/IB_SDP# LD_PRELOAD=libsdp.so perl my_client.pl rum-ib0
> 2009-07-07 19:07:08 my_client.pl 4300: recv hdr [25]: invalid magic:  32 30 30 39 2d 30 37 2d
> 
> which hapeens to be the hexdump of the string "2009-07-".
> where did it copy_user() that from? wtf?
> 
> root at kugel:/home/lars/DRBD/IB_SDP# LD_PRELOAD=libsdp.so perl my_client.pl rum-ib0
> 2009-07-07 19:07:08 my_client.pl 4301: recv payload [35]: expected 12296, but received 12295 byte; last bytes received:  55 e4 e3 e2
>                                  ^^^^ pid,           ^^ seq number.
> so for only 35 ping/pongs it did work ok.
> 
> exactly: one byte too short.
> the trailing magic expected is e4 e3 e1 e1
> 
> root at kugel:/home/lars/DRBD/IB_SDP# LD_PRELOAD=libsdp.so perl my_client.pl rum-ib0
> 2009-07-07 19:07:09 my_client.pl 4302: recv payload [33]: expected 4131, but received 4130 byte; last bytes received:  55 e4 e3 e2
> 
> root at kugel:/home/lars/DRBD/IB_SDP# LD_PRELOAD=libsdp.so perl my_client.pl rum-ib0
> 2009-07-07 19:07:10 my_client.pl 4303: recv payload [29]: expected 16401, but received 16400 byte; last bytes received:  55 e4 e3 e2
> 
> root at kugel:/home/lars/DRBD/IB_SDP# LD_PRELOAD=libsdp.so perl my_client.pl rum-ib0
> 2009-07-07 19:07:12 my_client.pl 4304: recv payload [21]: expected 4110, but received 4109 byte; last bytes received:  55 e4 e3 e2
> 
> root at kugel:/home/lars/DRBD/IB_SDP# LD_PRELOAD=libsdp.so perl my_client.pl rum-ib0
> 2009-07-07 19:07:13 my_client.pl 4305: recv payload [4]: expected 20495, but received 20494 byte; last bytes received:  55 e4 e3 e2
> 
> root at kugel:/home/lars/DRBD/IB_SDP# LD_PRELOAD=libsdp.so perl my_client.pl rum-ib0
> 2009-07-07 19:07:14 my_client.pl 4306: recv payload [12]: expected 22530, but received 22529 byte; last bytes received:  55 e4 e3 e2
> 
> 
> any suggestions how to proceed from here?
> 

-- 
Amir Vadai
Software Eng.
Mellanox Technologies
mailto: amirv at mellanox.co.il
Tel +972-3-6259539