[ewg] iWARP uDAPL RDMA_READ(RR) test hangs on Chelsio T320

[John Jolly]

On Thu, Jun 13, 2013 at 09:03:41AM -0500, Steve Wise wrote:
> It definitely isn't supported.  We should see some synchronous
> error, though, when posting the read work requests. IE
> dat_ep_post_rdma_read() should be failing.  It would be nice if you
> could verify this.

We get the following when enabling all debug levels:

# export DAPL_DBG_TYPE=0xffff
# dapltest -T P -s 10.0.0.1 -D ofa-v2-iwarp -d -i 10 RR 4096 2

...

Test[b0df]: Begin...
3650M3:10f8:5d4ea700: 42132 us(177 us): dapl_ep_post_rdma_read (0x63c0d0, 2, 0x63f8e0, (nil), 0x7fff51d7b200, 0)
3650M3:10f8:5d4ea700: 42137 us(5 us):  post_snd: ep 0x7f905c548a80 op 6537424 ck 0x4 sgs3650M3:10f8:5d4ea700: 42138 us(1 us):  post_snd: lkey 0x5ff va 0x63d700 len 4096
3650M3:10f8:5d4ea700: 42142 us(4 us):  post_snd: lkey 0x5ff va 0x63e700 len 4096
3650M3:10f8:5d4ea700: 42145 us(3 us):  post_snd_rdma: rkey 0xbff va 0x00000000645900
3650M3:10f8:5d4ea700: 42149 us(4 us):  post_snd: op 0x4 flags 0x2 sglist 0x63f8e0, 2
3650M3:10f8:5d4ea700: 42153 us(4 us): dapl_ep_post_rdma_read () returns 0x0
3650M3:10f8:5d4ea700: 42159 us(6 us): dapl_ep_post_rdma_read (0x63c0d0, 2, 0x63f8e0, (nil), 0x7fff51d7b200, 0)
3650M3:10f8:5d4ea700: 42164 us(5 us):  post_snd: ep 0x7f905c548a80 op 6537424 ck 0x4 sgs3650M3:10f8:5d4ea700: 42165 us(1 us):  post_snd: lkey 0x5ff va 0x63d700 len 4096
3650M3:10f8:5d4ea700: 42168 us(3 us):  post_snd: lkey 0x5ff va 0x63e700 len 4096
3650M3:10f8:5d4ea700: 42171 us(3 us):  post_snd_rdma: rkey 0xbff va 0x00000000645900
3650M3:10f8:5d4ea700: 42177 us(6 us):  post_snd: op 0x4 flags 0x2 sglist 0x63f8e0, 2
3650M3:10f8:5d4ea700: 42180 us(3 us): dapl_ep_post_rdma_read () returns 0x0
3650M3:10f8:5d4ea700: 42187 us(7 us): dapl_ep_post_rdma_read (0x63c0d0, 2, 0x63f8e0, (nil), 0x7fff51d7b200, 0)
3650M3:10f8:5d4ea700: 42191 us(4 us):  post_snd: ep 0x7f905c548a80 op 6537424 ck 0x4 sgs3650M3:10f8:5d4ea700: 42192 us(1 us):  post_snd: lkey 0x5ff va 0x63d700 len 4096
3650M3:10f8:5d4ea700: 42197 us(5 us):  post_snd: lkey 0x5ff va 0x63e700 len 4096
3650M3:10f8:5d4ea700: 42201 us(4 us):  post_snd_rdma: rkey 0xbff va 0x00000000645900
3650M3:10f8:5d4ea700: 42205 us(4 us):  post_snd: op 0x4 flags 0x2 sglist 0x63f8e0, 2
3650M3:10f8:5d4ea700: 42209 us(4 us): dapl_ep_post_rdma_read () returns 0x0
3650M3:10f8:5d4ea700: 42215 us(6 us): dapl_ep_post_rdma_read (0x63c0d0, 2, 0x63f8e0, (nil), 0x7fff51d7b200, 0)
3650M3:10f8:5d4ea700: 42219 us(4 us):  post_snd: ep 0x7f905c548a80 op 6537424 ck 0x4 sgs3650M3:10f8:5d4ea700: 42220 us(1 us):  post_snd: lkey 0x5ff va 0x63d700 len 4096
3650M3:10f8:5d4ea700: 42223 us(3 us):  post_snd: lkey 0x5ff va 0x63e700 len 4096
3650M3:10f8:5d4ea700: 42228 us(5 us):  post_snd_rdma: rkey 0xbff va 0x00000000645900
3650M3:10f8:5d4ea700: 42232 us(4 us):  post_snd: op 0x4 flags 0x2 sglist 0x63f8e0, 2
3650M3:10f8:5d4ea700: 42235 us(3 us): dapl_ep_post_rdma_read () returns 0x0
3650M3:10f8:5d4ea700: 42242 us(7 us): dapl_ep_post_rdma_read (0x63c0d0, 2, 0x63f8e0, (nil), 0x7fff51d7b200, 0)
3650M3:10f8:5d4ea700: 42247 us(5 us):  post_snd: ep 0x7f905c548a80 op 6537424 ck 0x4 sgs3650M3:10f8:5d4ea700: 42248 us(1 us):  post_snd: lkey 0x5ff va 0x63d700 len 4096
3650M3:10f8:5d4ea700: 42252 us(4 us):  post_snd: lkey 0x5ff va 0x63e700 len 4096
3650M3:10f8:5d4ea700: 42256 us(4 us):  post_snd_rdma: rkey 0xbff va 0x00000000645900
3650M3:10f8:5d4ea700: 42259 us(3 us):  post_snd: op 0x4 flags 0x2 sglist 0x63f8e0, 2
3650M3:10f8:5d4ea700: 42263 us(4 us): dapl_ep_post_rdma_read () returns 0x0
3650M3:10f8:5d4ea700: 42271 us(8 us): dapl_ep_post_rdma_read (0x63c0d0, 2, 0x63f8e0, (nil), 0x7fff51d7b200, 0)
3650M3:10f8:5d4ea700: 42275 us(4 us):  post_snd: ep 0x7f905c548a80 op 6537424 ck 0x4 sgs3650M3:10f8:5d4ea700: 42276 us(1 us):  post_snd: lkey 0x5ff va 0x63d700 len 4096
3650M3:10f8:5d4ea700: 42280 us(4 us):  post_snd: lkey 0x5ff va 0x63e700 len 4096
3650M3:10f8:5d4ea700: 42283 us(3 us):  post_snd_rdma: rkey 0xbff va 0x00000000645900
3650M3:10f8:5d4ea700: 42288 us(5 us):  post_snd: op 0x4 flags 0x2 sglist 0x63f8e0, 2
3650M3:10f8:5d4ea700: 42292 us(4 us): dapl_ep_post_rdma_read () returns 0x0
3650M3:10f8:5d4ea700: 42300 us(8 us): dapl_ep_post_rdma_read (0x63c0d0, 2, 0x63f8e0, (nil), 0x7fff51d7b200, 0)
3650M3:10f8:5d4ea700: 42304 us(4 us):  post_snd: ep 0x7f905c548a80 op 6537424 ck 0x4 sgs3650M3:10f8:5d4ea700: 42305 us(1 us):  post_snd: lkey 0x5ff va 0x63d700 len 4096
3650M3:10f8:5d4ea700: 42310 us(5 us):  post_snd: lkey 0x5ff va 0x63e700 len 4096
3650M3:10f8:5d4ea700: 42314 us(4 us):  post_snd_rdma: rkey 0xbff va 0x00000000645900
3650M3:10f8:5d4ea700: 42318 us(4 us):  post_snd: op 0x4 flags 0x2 sglist 0x63f8e0, 2
3650M3:10f8:5d4ea700: 42322 us(4 us): dapl_ep_post_rdma_read () returns 0x0
3650M3:10f8:5d4ea700: 42329 us(7 us): dapl_ep_post_rdma_read (0x63c0d0, 2, 0x63f8e0, (nil), 0x7fff51d7b200, 0)
3650M3:10f8:5d4ea700: 42333 us(4 us):  post_snd: ep 0x7f905c548a80 op 6537424 ck 0x4 sgs3650M3:10f8:5d4ea700: 42334 us(1 us):  post_snd: lkey 0x5ff va 0x63d700 len 4096
3650M3:10f8:5d4ea700: 42339 us(5 us):  post_snd: lkey 0x5ff va 0x63e700 len 4096
3650M3:10f8:5d4ea700: 42343 us(4 us):  post_snd_rdma: rkey 0xbff va 0x00000000645900
3650M3:10f8:5d4ea700: 42346 us(3 us):  post_snd: op 0x4 flags 0x2 sglist 0x63f8e0, 2
3650M3:10f8:5d4ea700: 42350 us(4 us): dapl_ep_post_rdma_read () returns 0x0
==> BLOCKING_MODE: DT_performance_wait()
3650M3:10f8:5d4ea700: 42368 us(18 us): dapl_evd_wait (0x63b3a0, -1, 1, 0x7fff51d7b180, 0x7fff51d7b174)
3650M3:10f8:5d4ea700: 42372 us(4 us): dapl_evd_wait: EVD 0x63b3a0, CQ 0x634ee0
3650M3:10f8:5d4ea700: 42379 us(7 us):  cq_object_wait: EVD 0x63b3a0 time -1

It appears to be returning 0

> 
> 
> >>We shouldn't see a hang though. :)
> >We are getting reports of other dapltest problems, such as
> >'!(t4_rq_empty(wq))' assertion on a pair of T422s. I have the feeling
> >I will become very familiar with dapltest.
> 
> There were some fixes in the flush logic of the t4 sw recently. They
> aren't upstream yet, but you can most likely get them from the
> latest T4 kit from chelsio's web site.
> 
> You should definitely send email to support at chelsio.com and get a
> problem report opened for this.
> 
> 
> Steve.
> 

Regards,
John