[ofa-general] Re: NOSRQ QP implementation issues

[Roland Dreier]

 > For sending (both on the active and passive side) the skbs are associated 
 > with the tx_qp. The remote qp for the tx_qp is the rx_qp (on the other side)
 > and WRs are posted to receive packets. An skb (for send) is not associated 
 > with SQ of the rx_qp. Therefore, no packets are expected to be sent through
 > the rx_qp.
 > 
 > In an erroneous case if packets do get sent to the wrong RQ, then they will
 > get dropped as no WQEs are posted. As discussed, an RNR will be returned as
 > expected and a new connection will get established. I still see no issues 
 > with this either.
 > 
 > If in the future, we do want to use the unused SQ and RQs, then we will have
 > to associate them with corresponding QP at the remote end. This will be work
 > for both the SRQ and non-SRQ case.
 > 
 > I do not see any issues. Can you please explain what is missing with this 
 > implementation?

I think what you are missing is that Linux is not necessarily the only
IPoIB CM implementation.  The Linux IPoIB driver needs to be able to
talk to any other implementation that follows the RFCs, in particular
RFC 4755 for connected mode.  And according to my reading of the RFC
at least, it is OK for a system to accept an IPoIB CM connection and
then use that connection to send packets back to the system that
originated the connection.  There is no requirement that a new
connection be opened for traffic in the other direction.

And killing the connection as soon as a packet is sent in the wrong
direction seems pretty wrong to me.  The current SRQ code actually
handles it fine, because all the QPs, no matter which direction they
were opened, are attached to the SRQ and hence have receives available.

One possibility would be to set the maxium receive MTU to 0 for
connections initiated in the no-SRQ case.  However I'm not sure
whether that is within the spirit of the RFC, and it might really
confuse other systems that do want to send on that QP.  Another
possibility would be to post one receive to all no-SRQ QPs, and if
that receive is consumed then post more.

 - R.