[ofa-general] Re: NOSRQ QP implementation issues

[Pradeep Satyanarayana]

Roland Dreier wrote:
>  > For sending (both on the active and passive side) the skbs are associated 
>  > with the tx_qp. The remote qp for the tx_qp is the rx_qp (on the other side)
>  > and WRs are posted to receive packets. An skb (for send) is not associated 
>  > with SQ of the rx_qp. Therefore, no packets are expected to be sent through
>  > the rx_qp.
>  > 
>  > In an erroneous case if packets do get sent to the wrong RQ, then they will
>  > get dropped as no WQEs are posted. As discussed, an RNR will be returned as
>  > expected and a new connection will get established. I still see no issues 
>  > with this either.
>  > 
>  > If in the future, we do want to use the unused SQ and RQs, then we will have
>  > to associate them with corresponding QP at the remote end. This will be work
>  > for both the SRQ and non-SRQ case.
>  > 
>  > I do not see any issues. Can you please explain what is missing with this 
>  > implementation?
> 
> I think what you are missing is that Linux is not necessarily the only
> IPoIB CM implementation.  The Linux IPoIB driver needs to be able to
> talk to any other implementation that follows the RFCs, in particular
> RFC 4755 for connected mode.  And according to my reading of the RFC
> at least, it is OK for a system to accept an IPoIB CM connection and
> then use that connection to send packets back to the system that
> originated the connection.  There is no requirement that a new
> connection be opened for traffic in the other direction.
> 
> And killing the connection as soon as a packet is sent in the wrong
> direction seems pretty wrong to me.  The current SRQ code actually
> handles it fine, because all the QPs, no matter which direction they
> were opened, are attached to the SRQ and hence have receives available.
> 
> One possibility would be to set the maxium receive MTU to 0 for
> connections initiated in the no-SRQ case.  However I'm not sure
> whether that is within the spirit of the RFC, and it might really
> confuse other systems that do want to send on that QP.  Another
> possibility would be to post one receive to all no-SRQ QPs, and if
> that receive is consumed then post more.
> 
>  - R.
> 
Thanks for pointing that out Roland. Yes, I was focussed on Linux and did not
consider other systems.

Michael, Thanks for catching this. Till I saw Roland's description I did not 
consider the other possibilities and did not see what you were alluding to.

What do you folks think about the following: in addition to posting 1 WR
suppose I create a separate CQ for the RQ (for tx_qp). There will be a small
completion handler that spits out a message that this request was received
from a non-Linux system, and then calls ipoib_ib_completion(). So, this way
we will not kill the connection, but the performance may be limited.

Pradeep