[openib-general] Re: Problems with dsp

[Ely Levy]

On Mon, 19 Dec 2005, Michael S. Tsirkin wrote:

> Quoting r. Ely Levy <elylevy at cs.huji.ac.il>:
> > Subject: Problems with dsp
> >
> > I'm trying to run iperf with sdp using svn from few days ago It works
> > find
> > until the speed goes above 1.49GB (by making block size bigger or
> > running
> > 2 iperf)
> > Then I'm starting to get weird debug messages in the log and as wall
> > from
> > root.
> > Any idea what might cause it?
> > This is what I see in /var/log/message:
> >
> > Dec 15 17:34:33 cmos-17 kernel: ib_sdp INIT: SDP module load.
> > Dec 15 17:34:33 cmos-17 kernel: ib_sdp INIT: Initializing /proc
> > filesystem
> > entries.
> > Dec 15 17:34:33 cmos-17 kernel: ib_sdp INIT: Advertisment cache
> > initialization.
> > Dec 15 17:34:33 cmos-17 kernel: ib_sdp INIT: Link level services
> > initialization.
> > Dec 15 17:34:33 cmos-17 kernel: ib_sdp INIT: Main pool initialized.
> > Dec 15 17:34:33 cmos-17 kernel: ib_sdp INIT: Creating connection tables.
> > Dec 15 17:34:33 cmos-17 kernel: ib_sdp INIT: IOCB cache initialization.
> > Dec 15 17:34:33 cmos-17 kernel: ib_sdp INIT: Started listening for SDP
> > connection requests
> > Dec 15 17:34:33 cmos-17 kernel: NET: Registered protocol family 27
> > Dec 15 17:34:47 cmos-17 kernel: ib_sdp CRTL: SOCKET: type <1> proto <0>
> > state <1:00000000>
> > Dec 15 17:34:47 cmos-17 kernel: ib_sdp CRTL: <0> <8e01> BIND: family <2>
> > addr <00000000:8913>
> > Dec 15 17:34:47 cmos-17 kernel: ib_sdp CRTL: <0> <8e01> LISTEN: addr
> > <00000000:1389> backlog <0005>
> > Dec 15 17:34:47 cmos-17 kernel: ib_sdp CRTL: <0> <0100> ACCEPT: addr
> > <00000000:1389>
> > Dec 15 17:34:51 cmos-17 kernel: ib_sdp CRTL: event <1> commID <00000014>
> > ID <-180208768>
> > Dec 15 17:34:51 cmos-17 kernel: ib_sdp CRTL: CM REQ. comm <00000014> SID
> > <8913010000000000> ca <mthca0> port <1>
> > Dec 15 17:34:51 cmos-17 kernel: ib_sdp CRTL: Hello BSDH
> > <003f:00:00:0000005c:00000000:00000000>
> > Dec 15 17:34:51 cmos-17 kernel: ib_sdp CRTL: Hello HH
> > <ff:40:11:00001000:00001000:8001:0a000701:0a000702>
> > Dec 15 17:34:51 cmos-17 kernel: ib_sdp CRTL: <0> <0100> ACCEPT: complete
> > <1> <0a000702:1389><0a000701:8001>
> > Dec 15 17:34:51 cmos-17 kernel: ib_sdp CRTL: <1> <2340> GETNAME: src
> > <0a000702:1389> dst <0a000701:8001>
> > Dec 15 17:34:51 cmos-17 kernel: ib_sdp CRTL: <1> <2340> GETNAME: src
> > <0a000702:1389> dst <0a000701:8001>
> > Dec 15 17:34:51 cmos-17 kernel: ib_sdp CRTL: event <4> commID <00000014>
> > ID <1>
> > Dec 15 17:34:51 cmos-17 kernel: ib_sdp CRTL: <1> <2340> CM ESTABLISHED.
> > commID <00000014>
> > Dec 15 17:34:51 cmos-17 kernel: ib_sdp CRTL: <1> <1171> Passive
> > Establish
> > src <0a000702:1389> dst <0a000701:8001>
> > Dec 15 17:34:51 cmos-17 kernel: ib_sdp CRTL: <1> <1171> Mode request <2>
> > from current mode. <1:1>
> > Dec 15 17:34:51 cmos-17 kernel: ib_sdp CRTL: <0> <0100> ACCEPT: addr
> > <00000000:1389>
> > Dec 15 17:34:51 cmos-17 kernel: <7shent <1>
> > Dec 15 17:34:51 cmos-17 kernel: 6> bytes.
> > Dec 15 17:34:51 cmos-17 kernel: < wrid <2543> of <4096> bytplete1171>
> > RECV
> > BUFF, bytes <4096>
> > Dec 15 17:34:51 cmos-17 kernel: <1171> Read complete <dp DATA: <POST
> > READ
> > BUFF w1171> Read complete <3888> of <4096> bytes.
> > Dec 15 17:34:51 cmos-17 kernel: POST READ BUFF wrid <3959> of <4096>
> > bytes.
> > Dec 15 17:34:51 cmos-17 kernel: <<1> <1171> POST READ BUFF wrid <4186>
> > of
> > <4096> bytes.
> > Dec 15 17:34:51 cmos-17 kernel: <f <4096> bytes.
> > Dec 15 17:34:51 cmos-17 kernel: 357> of <4096> bytes.
> > Dec 15 17:34:51 cmos-17 kernel: <7ST READ BUFF wrid71> Read complete
> > <4296
> > DATA: <1>71> POST READib_sdp DATA: <1> <ib_sdp DATA:ib_sdp DATA: <1>
> > <1171> POST READ BUFF wrid <4360> of <4096> bytes.
> > Dec 15 17:34:51 cmos-17 kernel: <7id <5116> of <4096> bytes.te 1> RECV
> > BUFF,te <5066> of <4096> byt1> RECV BUFF, bytes <
> > UFF wr> <1171> Read>> <1171> PTA: <1> <1171> Re.
> > Dec 15 17:34:51 cmos-17 kernel: <7 <1> <1171> Read complete <5507> of
> > <4096> byte <1> <1171> RECV BUFF, bytes <4096>
> > Dec 15 17:34:51 cmos-17 kernel:  bytes.
> > Dec 15 17:34:51 cmos-17 kernel: <_sdp DATA: <1>  <4096> bytes.
> > Dec 15 17:34:51 cmos-17 kernel: <7 <40, bytes <4096>
> > Dec 15 17:34:51 cmos-17 kernel: 63> of <40<4096>
> > Dec 15 17:34:51 cmos-17 kernel: <7te <6164> of <4096> bytes.
> > Dec 15 17:34:51 cmos-17 kernel: 5> of <4096> bytes.
> > Dec 15 17:34:51 cmos-17 kernel: <74096>
> > Dec 15 17:34:51 cmos-17 kernel: <7d <6450> of <409e <6396> of <4096>
> > bytes.
> >
>
> Are you saying you see these when SDP is build with debug disabled?

No, DSP debug is enabled but it doesn't start sending those messages as
wall unless you get to 1.49GB speed, it also seems to stop at that speed
though I thought infiniband can get to higher bw.
Also as you can see the debuging info is very messed up so it's hard to
tell what was going on.
> --
> MST
>

Ely