[ofa-general] disconnect implementation for rdma cm unconnected datagram service

[Sean Hefty]

Or Gerlitz wrote:
> Looking on cm_sidr_rep_handler we see that the cm id state
> is reseted to IB_CM_IDLE, and on the other hand ib_send_cm_dreq
> returns -EINVAL if the id state is not IB_CM_ESTABLISHED. I gueess
> this means that rdma_disconnect on RDMA_PS_UDP would never work?

Correct - there isn't a disconnect for UDP.

> Thinking on remote qp/lid change, the equivalent I see for UDP based apps,
> is that a remote qp/lid change would have been caught by the local stack
> neighbouring system since it sends few unicast arps probes and the re-issues
> a broadcast arp from which the new HW address (qpn / gid --> lid) would be learned.
> 
> What you think would be the correct way to solve that for rdmacm based apps?

I don't know that we can do anything about a QP change.

> is there a way for the RDMA/IB stack level to provide the solution? we were

Once the inform_info patches are in, we might be able to hook into that 
to at least provide notification that the remote address has changed.  I 
don't think there's a LID change notice, though, only GID IN/OUT.  LID 
changes would be difficult to hide from the app anyway, since the app 
must re-create their address vector.

If we ever go as far as adding an rdma_send() call, we might be able to 
hide it better.

> I guess that remote lid change can be emulated as disconnect if the rdmacm
> would listen on IN/OUT traps, but the question if what can we do about the
> remote process qp, eg in the case the process dies and then comes back again etc.

I think the current solution is that the app must detect that they are 
no longer getting responses from the remote side and try to 
re-'connect'.  I need to give this more thought to determine if there's 
anything that we can do here.  (This seems hard without the rdma_cm 
controlling the QP and CQs.)  Do you have any ideas?

- Sean