[openib-general] Re: [uDAPL] dtest server never ends when using the dapl provider "OpenIB-scm1"

Wed Apr 12 01:01:43 PDT 2006

On Tuesday 11 April 2006 20:45, Arlin Davis wrote:
> James Lentini wrote:
> 
> >  
> >
> > It sounds like the disconnect is being lost. Let me see if I can
> >
> >reproduce this.
> >
> >Arlin, have you ever seen this?
> >  
> >
> No. it runs fine on my systems. It looks like the ping pong test on the 
> server side did not finish. Can Dotan add a -v switch to the dtest to 
> help isolate? 
> 
> What svn version are we running? Do you have the latest uDAPL fixes 
> commited in 6393?
> 
> -arlin
> 
> 

Hi.

thanks for the quick response.

I executed the dtest with the -v parameter and here is the output of both sides.
I added the test the '-l' parameter to be able to change to dapl provider in command line (if you wish i can post you a patch).

full server output:
-----------------------
sw043:/tmp/tsscr/svn.mlx_tp/gen2/userspace/ulps/udapl/dtest # ./dtest -l OpenIB-scm2 -v
23996 DAPL_PROVIDER is OpenIB-scm2
23996 Verbose
23996 Running as server
23996 Allocated RDMA buffers (r:0x8052390,s:0x8052618) len 64
23996 Opened Interface Adaptor
23996 Create Protection Zone
23996 Created Protection Zone
23996 Register RDMA memory
23996 Registered Receive RDMA Buffer 0x8052390
23996 Registered Send RDMA Buffer 0x8052618
23996 Register RDMA memory done
23996 Create events
23996 cr_evd created 0x805d6f8
23996 con_evd created 0x805d940
23996 dto_req_evd created 0x805dc18
23996 dto_rcv_evd created 0x805deb8
23996 Create events done
23996 EP created 0x805f518
23996 Registering send Message Buffer 0x804f6d0, len 24
23996 Registered send Message Buffer 0x804f6d0
23996 Registering Receive Message Buffer 0x804f700
23996 Registered Receive Message Buffer 0x804f700
23996 Posting Receive Message Buffer 0x804f700
23996 Registered Receive Message Buffer 0x804f700
23996 Posting Receive Message Buffer 0x804f718
23996 Registered Receive Message Buffer 0x804f700
23996 Posting Receive Message Buffer 0x804f730
23996 Registered Receive Message Buffer 0x804f700
23996 Creating service point for listen
23996 dat_psp_created for server listen
23996 Server waiting for connect request..
23996 dat_evd_wait for cr_evd completed
23996 Accepting connect request from client
23996 dat_cr_accept completed
23996 Waiting for connect response
23996 dat_evd_wait for h_conn_evd completed

23996 CONNECTED!

23996 Send RMR to remote: snd_msg: r_key_ctx=620439,pad=0,va=8052390,len=0x40
23996 calling post_send
23996 send_msg completed
23996 Waiting for remote to send RMR data
23996 dat_evd_wait h_dto_rcv_evd completed
23996 remote RMR data arrived!
23996 Received RMR from remote: r_iov: r_key_ctx=620436,pad=0,va=8051600,len=0x40
23996 connect_ep complete

 23996 RDMA WRITE DATA with SEND MSG

23996 rdma_write # 1 completed
23996 rdma_write # 2 completed
23996 rdma_write # 3 completed
23996 rdma_write # 4 completed
23996 rdma_write # 5 completed
23996 rdma_write # 6 completed
23996 rdma_write # 7 completed
23996 rdma_write # 8 completed
23996 rdma_write # 9 completed
23996 rdma_write # 10 completed
23996 Sending completion message
23996 calling post_send
23996 send_msg completed
23996 waiting for message receive event
23996 inbound rdma_write; send message arrived!
23996 Received RMR from remote: r_iov: ctx=620436,pad=0,va=0x8051600,len=0x40
23996 inbound rdma_write; send msg event SUCCESS!!!
23996 SERVER: RDMA write buffer contains: client written data...
23996 do_rdma_write_with_msg complete

 23996 RDMA READ DATA with SEND MSG

23996 waiting for rdma_read completion event
23996 rdma_read # 1 completed
23996 waiting for rdma_read completion event
23996 rdma_read # 2 completed
23996 waiting for rdma_read completion event
23996 rdma_read # 3 completed
23996 waiting for rdma_read completion event
23996 rdma_read # 4 completed
23996 Sending completion message
23996 calling post_send
23996 send_msg completed
23996 Waiting for inbound message....
23996 waiting for message receive event
23996 inbound rdma_read; send message arrived!
23996 Received RMR from remote: r_iov: ctx=620436,pad=0,va=0x8051600,len=0x40
23996 inbound rdma_write; send msg event SUCCESS!!!
23996 SERVER: RCV RDMA read buffer contains: server written data...
23996 do_rdma_read_with_msg complete

 23996 PING DATA with SEND MSG

23996 Pre-posting Receive Message Buffers 0x8052390
23996 Posted Receive Message Buffer 0x8052390
23996 Pre-posting Receive Message Buffers 0x80523d0
23996 Posted Receive Message Buffer 0x80523d0
23996 Pre-posting Receive Message Buffers 0x8052410
23996 Posted Receive Message Buffer 0x8052410
23996 Pre-posting Receive Message Buffers 0x8052450
23996 Posted Receive Message Buffer 0x8052450
23996 Pre-posting Receive Message Buffers 0x8052490
23996 Posted Receive Message Buffer 0x8052490
23996 Pre-posting Receive Message Buffers 0x80524d0
23996 Posted Receive Message Buffer 0x80524d0
23996 Pre-posting Receive Message Buffers 0x8052510
23996 Posted Receive Message Buffer 0x8052510
23996 Pre-posting Receive Message Buffers 0x8052550
23996 Posted Receive Message Buffer 0x8052550
23996 Pre-posting Receive Message Buffers 0x8052590
23996 Posted Receive Message Buffer 0x8052590
23996 Pre-posting Receive Message Buffers 0x80525d0
23996 Posted Receive Message Buffer 0x80525d0
23996 waiting for message receive event
23996 inbound message; message arrived!
23996 SERVER: RCV buffer 0x8052390 contains: 0x55 len=64
23996 SERVER: SND buffer 0x8052618 contains: 0xffffffaa len=64
23996 calling post_send
23996 send_msg completed
23996 waiting for message receive event
23996 inbound message; message arrived!
23996 SERVER: RCV buffer 0x80523d0 contains: 0x55 len=64
23996 SERVER: SND buffer 0x8052658 contains: 0xffffffaa len=64
23996 calling post_send
23996 send_msg completed
23996 waiting for message receive event
23996 inbound message; message arrived!
23996 SERVER: RCV buffer 0x8052410 contains: 0x55 len=64
23996 SERVER: SND buffer 0x8052698 contains: 0xffffffaa len=64
23996 calling post_send
23996 send_msg completed
23996 waiting for message receive event
23996 inbound message; message arrived!
23996 SERVER: RCV buffer 0x8052450 contains: 0x55 len=64
23996 SERVER: SND buffer 0x80526d8 contains: 0xffffffaa len=64
23996 calling post_send
23996 send_msg completed
23996 waiting for message receive event
23996 inbound message; message arrived!
23996 SERVER: RCV buffer 0x8052490 contains: 0x55 len=64
23996 SERVER: SND buffer 0x8052718 contains: 0xffffffaa len=64
23996 calling post_send
23996 send_msg completed
23996 waiting for message receive event
23996 inbound message; message arrived!
23996 SERVER: RCV buffer 0x80524d0 contains: 0x55 len=64
23996 SERVER: SND buffer 0x8052758 contains: 0xffffffaa len=64
23996 calling post_send
23996 send_msg completed
23996 waiting for message receive event
23996 inbound message; message arrived!
23996 SERVER: RCV buffer 0x8052510 contains: 0x55 len=64
23996 SERVER: SND buffer 0x8052798 contains: 0xffffffaa len=64
23996 calling post_send
23996 send_msg completed
23996 waiting for message receive event
23996 inbound message; message arrived!
23996 SERVER: RCV buffer 0x8052550 contains: 0x55 len=64
23996 SERVER: SND buffer 0x80527d8 contains: 0xffffffaa len=64
23996 calling post_send
23996 send_msg completed
23996 waiting for message receive event
23996 inbound message; message arrived!
23996 SERVER: RCV buffer 0x8052590 contains: 0x55 len=64
23996 SERVER: SND buffer 0x8052818 contains: 0xffffffaa len=64
23996 calling post_send
23996 send_msg completed
23996 waiting for message receive event
23996 inbound message; message arrived!
23996 SERVER: RCV buffer 0x80525d0 contains: 0x55 len=64
23996 SERVER: SND buffer 0x8052858 contains: 0xffffffaa len=64
23996 calling post_send
23996 send_msg completed
23996 do_ping_pong_msg complete
23996 Disconnect and Free EP 0x805f518

full client output
-----------------------
sw046:/tmp/tsscr/svn.mlx_tp/gen2/userspace/ulps/udapl/dtest # ./dtest -h 12.4.3.43 -l OpenIB-scm2 -v
30681 DAPL_PROVIDER is OpenIB-scm2
30681 Verbose
30681 Running as client
30681 Allocated RDMA buffers (r:0x8051600,s:0x8051888) len 64
30681 Opened Interface Adaptor
30681 Create Protection Zone
30681 Created Protection Zone
30681 Register RDMA memory
30681 Registered Receive RDMA Buffer 0x8051600
30681 Registered Send RDMA Buffer 0x8051888
30681 Register RDMA memory done
30681 Create events
30681 cr_evd created 0x8058708
30681 con_evd created 0x80589e0
30681 dto_req_evd created 0x8058cb8
30681 dto_rcv_evd created 0x805a3f0
30681 Create events done
30681 EP created 0x805a570
30681 Registering send Message Buffer 0x804f6d0, len 24
30681 Registered send Message Buffer 0x804f6d0
30681 Registering Receive Message Buffer 0x804f700
30681 Registered Receive Message Buffer 0x804f700
30681 Posting Receive Message Buffer 0x804f700
30681 Registered Receive Message Buffer 0x804f700
30681 Posting Receive Message Buffer 0x804f718
30681 Registered Receive Message Buffer 0x804f700
30681 Posting Receive Message Buffer 0x804f730
30681 Registered Receive Message Buffer 0x804f700
30681 Server Name: 12.4.3.43
30681 Server Net Address: 12.4.3.43
30681 Connecting to server
30681 dat_ep_connect completed
30681 Waiting for connect response
30681 dat_evd_wait for h_conn_evd completed

30681 CONNECTED!

30681 Send RMR to remote: snd_msg: r_key_ctx=620436,pad=0,va=8051600,len=0x40
30681 calling post_send
30681 send_msg completed
30681 Waiting for remote to send RMR data
30681 dat_evd_wait h_dto_rcv_evd completed
30681 remote RMR data arrived!
30681 Received RMR from remote: r_iov: r_key_ctx=620439,pad=0,va=8052390,len=0x40
30681 connect_ep complete

 30681 RDMA WRITE DATA with SEND MSG

30681 rdma_write # 1 completed
30681 rdma_write # 2 completed
30681 rdma_write # 3 completed
30681 rdma_write # 4 completed
30681 rdma_write # 5 completed
30681 rdma_write # 6 completed
30681 rdma_write # 7 completed
30681 rdma_write # 8 completed
30681 rdma_write # 9 completed
30681 rdma_write # 10 completed
30681 Sending completion message
30681 calling post_send
30681 send_msg completed
30681 waiting for message receive event
30681 inbound rdma_write; send message arrived!
30681 Received RMR from remote: r_iov: ctx=620439,pad=0,va=0x8052390,len=0x40
30681 inbound rdma_write; send msg event SUCCESS!!!
30681 CLIENT: RDMA write buffer contains: server written data...
30681 do_rdma_write_with_msg complete

 30681 RDMA READ DATA with SEND MSG

30681 waiting for rdma_read completion event
30681 rdma_read # 1 completed
30681 waiting for rdma_read completion event
30681 rdma_read # 2 completed
30681 waiting for rdma_read completion event
30681 rdma_read # 3 completed
30681 waiting for rdma_read completion event
30681 rdma_read # 4 completed
30681 Sending completion message
30681 calling post_send
30681 send_msg completed
30681 Waiting for inbound message....
30681 waiting for message receive event
30681 inbound rdma_read; send message arrived!
30681 Received RMR from remote: r_iov: ctx=620439,pad=0,va=0x8052390,len=0x40
30681 inbound rdma_write; send msg event SUCCESS!!!
30681 CLIENT: RCV RDMA read buffer contains: server read data...
30681 do_rdma_read_with_msg complete

 30681 PING DATA with SEND MSG

30681 Pre-posting Receive Message Buffers 0x8051600
30681 Posted Receive Message Buffer 0x8051600
30681 Pre-posting Receive Message Buffers 0x8051640
30681 Posted Receive Message Buffer 0x8051640
30681 Pre-posting Receive Message Buffers 0x8051680
30681 Posted Receive Message Buffer 0x8051680
30681 Pre-posting Receive Message Buffers 0x80516c0
30681 Posted Receive Message Buffer 0x80516c0
30681 Pre-posting Receive Message Buffers 0x8051700
30681 Posted Receive Message Buffer 0x8051700
30681 Pre-posting Receive Message Buffers 0x8051740
30681 Posted Receive Message Buffer 0x8051740
30681 Pre-posting Receive Message Buffers 0x8051780
30681 Posted Receive Message Buffer 0x8051780
30681 Pre-posting Receive Message Buffers 0x80517c0
30681 Posted Receive Message Buffer 0x80517c0
30681 Pre-posting Receive Message Buffers 0x8051800
30681 Posted Receive Message Buffer 0x8051800
30681 Pre-posting Receive Message Buffers 0x8051840
30681 Posted Receive Message Buffer 0x8051840
30681 CLIENT: SND buffer 0x8051888 contains: 0x55 len=64
30681 calling post_send
30681 send_msg completed
30681 waiting for message receive event
30681 inbound message; message arrived!
30681 CLIENT: RCV buffer 0x8051600 contains: 0xffffffaa len=64
30681 CLIENT: SND buffer 0x80518c8 contains: 0x55 len=64
30681 calling post_send
30681 send_msg completed
30681 waiting for message receive event
30681 inbound message; message arrived!
30681 CLIENT: RCV buffer 0x8051640 contains: 0xffffffaa len=64
30681 CLIENT: SND buffer 0x8051908 contains: 0x55 len=64
30681 calling post_send
30681 send_msg completed
30681 waiting for message receive event
30681 inbound message; message arrived!
30681 CLIENT: RCV buffer 0x8051680 contains: 0xffffffaa len=64
30681 CLIENT: SND buffer 0x8051948 contains: 0x55 len=64
30681 calling post_send
30681 send_msg completed
30681 waiting for message receive event
30681 inbound message; message arrived!
30681 CLIENT: RCV buffer 0x80516c0 contains: 0xffffffaa len=64
30681 CLIENT: SND buffer 0x8051988 contains: 0x55 len=64
30681 calling post_send
30681 send_msg completed
30681 waiting for message receive event
30681 inbound message; message arrived!
30681 CLIENT: RCV buffer 0x8051700 contains: 0xffffffaa len=64
30681 CLIENT: SND buffer 0x80519c8 contains: 0x55 len=64
30681 calling post_send
30681 send_msg completed
30681 waiting for message receive event
30681 inbound message; message arrived!
30681 CLIENT: RCV buffer 0x8051740 contains: 0xffffffaa len=64
30681 CLIENT: SND buffer 0x8051a08 contains: 0x55 len=64
30681 calling post_send
30681 send_msg completed
30681 waiting for message receive event
30681 inbound message; message arrived!
30681 CLIENT: RCV buffer 0x8051780 contains: 0xffffffaa len=64
30681 CLIENT: SND buffer 0x8051a48 contains: 0x55 len=64
30681 calling post_send
30681 send_msg completed
30681 waiting for message receive event
30681 inbound message; message arrived!
30681 CLIENT: RCV buffer 0x80517c0 contains: 0xffffffaa len=64
30681 CLIENT: SND buffer 0x8051a88 contains: 0x55 len=64
30681 calling post_send
30681 send_msg completed
30681 waiting for message receive event
30681 inbound message; message arrived!
30681 CLIENT: RCV buffer 0x8051800 contains: 0xffffffaa len=64
30681 CLIENT: SND buffer 0x8051ac8 contains: 0x55 len=64
30681 calling post_send
30681 send_msg completed
30681 waiting for message receive event
30681 inbound message; message arrived!
30681 CLIENT: RCV buffer 0x8051840 contains: 0xffffffaa len=64
30681 do_ping_pong_msg complete
30681 Disconnect and Free EP 0x805a570
30681 dat_ep_disconnect
30681 dat_ep_disconnect completed
30681 dat_evd_wait for h_conn_evd completed
30681 Unregister send message h_lmr 0x805c738
30681 Unregistered send message Buffer
30681 Unregister recv message h_lmr 0x805c800
30681 Unregistered recv message Buffer
30681 Free EP 0x805a570
30681 Freed EP
30681 destroy events
30681 Free cr EVD 0x8058708
30681 Freed cr EVD
30681 Free conn EVD 0x80589e0
30681 Freed conn EVD
30681 Free RCV dto EVD 0x805a3f0
30681 Freed dto EVD
30681 Free REQ dto EVD 0x8058cb8
30681 Freed dto EVD
30681 destroy events done
30681 Unregister h_lmr 0x80582b8
30681 Unregistered Recv Buffer
30681 Unregister h_lmr 0x8058658
30681 Unregistered send Buffer
30681 unregister_rdma_memory
30681 unregister_rdma_memory done
30681 Freeing pz
30681 Freed pz
30681 Closing Interface Adaptor
30681 Closed Interface Adaptor

30681: DAPL Test Complete.

30681: Message RTT: Total=   1160.92 usec, 10 bursts, itime=    116.09 usec, pc=0
30681: RDMA write:  Total=    892.01 usec, 10 bursts, itime=     89.20 usec, pc=0
30681: RDMA read:   Total=    244.90 usec,   4 bursts, itime=     79.05 usec, pc=0
30681: RDMA read:   Total=    244.90 usec,   4 bursts, itime=     55.01 usec, pc=0
30681: RDMA read:   Total=    244.90 usec,   4 bursts, itime=     55.95 usec, pc=0
30681: RDMA read:   Total=    244.90 usec,   4 bursts, itime=     54.89 usec, pc=0
30681: open:        49977.00 usec
30681: close:       71868.94 usec
30681: PZ create:      96.11 usec
30681: PZ free:        14.01 usec
30681: LMR create:    232.01 usec
30681: LMR free:       95.09 usec
30681: EVD create:     21.01 usec
30681: EVD free:      142.07 usec
30681: EP create:     496.94 usec
30681: EP free:       312.04 usec
30681: TOTAL:        1299.16 usec

I'm using driver IBED-1.0-rc3, openib_branch1.0-20060410-1551 (REV=6367)

when i tried to execute this test on openib_gen2-20060410-1700 (REV=6369), i got the same result.
when i tried to execute this test on openib_gen2-20060411-0800 (REV=6400), i got the same result.

thanks
Dotan