[libfabric-users] rdma fail with verbs

[Biddiscombe, John A.]

Sean, sorry your message went into my spam folder so I missed it last night

rping output

Client greina7 (192.168.3.37), server greina6 (192.168.3.36)

greina7:~/build/hvtkm$ rping -c -d -a 192.168.3.36 -p 7910 -S 32768 -C 5
port 7910
size 32768
count 5
created cm_id 0x60f6f0
cma_event type RDMA_CM_EVENT_ADDR_RESOLVED cma_id 0x60f6f0 (parent)
cma_event type RDMA_CM_EVENT_ROUTE_RESOLVED cma_id 0x60f6f0 (parent)
rdma_resolve_addr - rdma_resolve_route successful
created pd 0x60c100
created channel 0x60fa00
created cq 0x60c1a0
created qp 0x60fa48
rping_setup_buffers called on cb 0x6072e0
allocated & registered buffers...
cq_thread started.
cma_event type RDMA_CM_EVENT_ESTABLISHED cma_id 0x60f6f0 (parent)
ESTABLISHED
rmda_connect successful
RDMA addr 621030 rkey 64f2 len 32768
send completion
recv completion
RDMA addr 619020 rkey 56e5 len 32768
send completion
recv completion
RDMA addr 621030 rkey 64f2 len 32768
send completion
recv completion
RDMA addr 619020 rkey 56e5 len 32768
send completion
recv completion
RDMA addr 621030 rkey 64f2 len 32768
send completion
recv completion
RDMA addr 619020 rkey 56e5 len 32768
send completion
recv completion
RDMA addr 621030 rkey 64f2 len 32768
send completion
recv completion
RDMA addr 619020 rkey 56e5 len 32768
send completion
recv completion
RDMA addr 621030 rkey 64f2 len 32768
send completion
recv completion
RDMA addr 619020 rkey 56e5 len 32768
send completion
recv completion
rping_free_buffers called on cb 0x6072e0
cma_event type RDMA_CM_EVENT_DISCONNECTED cma_id 0x60f6f0 (parent)
client DISCONNECT EVENT...
destroy cm_id 0x60f6f0


greina6:~/build/hvtkm$ rping -d -s -p 7910 -S 32768 -C 5
server
port 7910
size 32768
count 5
created cm_id 0x60f640
rdma_bind_addr successful
rdma_listen
cma_event type RDMA_CM_EVENT_CONNECT_REQUEST cma_id 0x2aaab0000a30 (child)
child cma 0x2aaab0000a30
created pd 0x60c050
created channel 0x60f950
created cq 0x60c0f0
created qp 0x60f9c8
rping_setup_buffers called on cb 0x6072e0
allocated & registered buffers...
accepting client connection request
cq_thread started.
cma_event type RDMA_CM_EVENT_ESTABLISHED cma_id 0x2aaab0000a30 (child)
ESTABLISHED
recv completion
Received rkey 64f2 addr 621030 len 32768 from peer
server received sink adv
server posted rdma read req
rdma read completion
server received read complete
server posted go ahead
send completion
recv completion
Received rkey 56e5 addr 619020 len 32768 from peer
server received sink adv
rdma write from lkey 17cec8 laddr 619020 len 32768
rdma write completion
server rdma write complete
server posted go ahead
send completion
recv completion
Received rkey 64f2 addr 621030 len 32768 from peer
server received sink adv
server posted rdma read req
rdma read completion
server received read complete
server posted go ahead
send completion
recv completion
Received rkey 56e5 addr 619020 len 32768 from peer
server received sink adv
rdma write from lkey 17cec8 laddr 619020 len 32768
rdma write completion
server rdma write complete
server posted go ahead
send completion
recv completion
Received rkey 64f2 addr 621030 len 32768 from peer
server received sink adv
server posted rdma read req
rdma read completion
server received read complete
server posted go ahead
send completion
recv completion
Received rkey 56e5 addr 619020 len 32768 from peer
server received sink adv
rdma write from lkey 17cec8 laddr 619020 len 32768
rdma write completion
server rdma write complete
server posted go ahead
send completion
recv completion
Received rkey 64f2 addr 621030 len 32768 from peer
server received sink adv
server posted rdma read req
rdma read completion
server received read complete
server posted go ahead
send completion
recv completion
Received rkey 56e5 addr 619020 len 32768 from peer
server received sink adv
rdma write from lkey 17cec8 laddr 619020 len 32768
rdma write completion
server rdma write complete
server posted go ahead
send completion
recv completion
Received rkey 64f2 addr 621030 len 32768 from peer
server received sink adv
server posted rdma read req
rdma read completion
server received read complete
server posted go ahead
send completion
recv completion
Received rkey 56e5 addr 619020 len 32768 from peer
server received sink adv
rdma write from lkey 17cec8 laddr 619020 len 32768
rdma write completion
server rdma write complete
server posted go ahead
send completion
cma_event type RDMA_CM_EVENT_DISCONNECTED cma_id 0x2aaab0000a30 (child)
server DISCONNECT EVENT...
wait for RDMA_READ_ADV state 10
rping_free_buffers called on cb 0x6072e0
destroy cm_id 0x60f640