[ofa-general] Non-Signaled RDMA Read (cxgb3)

[Philip Frey1]

general-bounces at lists.openfabrics.org wrote on 29.07.2008 18:32:27:

> [image removed] 
> 
> Re: [ofa-general] Non-Signaled RDMA Read (cxgb3)
> 
> Steve Wise 
> 
> to:
> 
> Roland Dreier
> 
> 29.07.2008 18:32
> 
> Sent by:
> 
> general-bounces at lists.openfabrics.org
> 
> Cc:
> 
> Philip Frey1, general
> 
> Steve Wise wrote:
> > Roland Dreier wrote:
> >>  > I am trying to post a non-signaled RDMA Read WR. Somehow OFED 1.3 
> >> running  > on a T3 Chelsio Adapter seems
> >>  > not to care what I set for 'enum ibv_send_flags' in my 'struct  > 
> >> ibv_send_wr'. I always get a work completion with status
> >>  > set to IBV_WC_SUCCESS and opcode IBV_WC_RDMA_READ. When creating 
> >> the  > 'struct ibv_qp' I have set
> >>  > 'sq_sig_all = 0' in 'struct ibv_qp_init_attr'. With RDMA Write and 

> >> Send I  > do not get a WC when I set the 'ibv_send_flags'
> >>  > to 0.
> >>  >  > What am I missing here?
> >>
> >> Looks like you're hitting this -- the cxgb3 driver has:
> >>
> >>                 case IB_WR_RDMA_READ:
> >>                         t3_wr_opcode = T3_WR_READ;
> >>                         t3_wr_flags = 0; /* T3 reads are always 
> >> signaled */
> >>
> >> so I guess the HW can't do unsignaled RDMA reads.
> >>
> >>  - R.
> >> 
> > Right.  This is a T3 hw limitation.  It will be fixed in the next 
chip.
> >
> > Steve.
> >
> >
> Note that I _could_ add code in the driver/lib poll routine to "eat" 
> these cqes.  But there is a fundamental problem in that if the CQ is 
> armed, then the HW _will_ generate an interrupt in this case.  This 
> leads to a "false notification" which breaks the semantics of the 
> notification mechanism.  I'm not sure how bad that is?  To the 
> application, it would receive a notification, then go poll and get 
nothing.
> 
> Steve.

I do not think this will help. When I do a send, I am expecting a 
completion with
the matching work request ID and I do not want to add extra code into the 
upper layer
app to get rid of those false notifications (pitfall).

Speaking about this notification mechanism: What is the intended way of 
using that?
Say I expect a number of inbound sends. What disturbs me is that only the 
first of them
generates an event. I cannot just do an ibv_get_cq_event() after having 
gotten the first one.
It might happen that already several inbound sends have arrived so I need 
to poll the CQ
for all of them. Only when the polling fails I go to sleep waiting for an 
event. From a
user perspective it would be way more convenient to just call 
ibv_get_cq_event() several times.
Another disturbing thing is the fact that the events have to be 
acknowledged - why bother the user
with that?

If I may, I would like to suggest another thing: Is there no possibility 
that non-signaled WRs
do not create a WC at all? Clearing the SQ with a signaled WR to prevent 
an overflow is rather
cumbersome as the user has to keep track of the number of unsignaled WRs 
posted as well as the
size of the SQ. Things like that add a lot of pitfalls to upper layer 
applications.

Thanks,
 Philip
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20080729/ea1a4489/attachment.html>