[ofa-general] [PATCH RFC] librdmacm: implement RDMA_CM_EVENT_ADDR_CHANGE notification

[Or Gerlitz]

RDMA_CM_EVENT_ADDR_CHANGE event can be used by librdmacm consumers that wish
to have their RDMA sessions always use the same links (eg <hca/port>) as the
IP stack does. In the current code, this does not happen when bonding is used
and fail-over happened, but the IB link used by an already existing session is
operating fine.

The kernel rdma-cm code was enhanced to use netevent notification for sensing that
a change has happened in the IP stack, and deliver this event for ID that is misaligned
that respect with the IP stack. The user can act on the event or just ignore it

Signed-off-by: Or Gerlitz <ogerlitz at voltaire.com>

Sean,

This is the patch to librdmacm that brings RDMA_CM_EVENT_ADDR_CHANGE to
user space, please let me know your thoughts. The net-core/bonding patches
on which this patch set is depedent upon where pushed by the bonding
maintainer (Jay Vosburgh) to the netdev maintainer (Jeff Garzik).

Assuming Jeff accepts them into his tree, does it makes sense to you to pull
his tree into yours and then push the patch sequence to Roland, or you
want me to push it to Roland and you will sign/ack the patches? We are now
at -rc6 and I want to bring this to the for-2.6.27 branch of Roland's this week.

Roland - is there an easier way to do this push?

Or.

---
 examples/rping.c        |    4 ++++
 include/rdma/rdma_cma.h |    3 ++-
 man/rdma_get_cm_event.3 |    5 +++++
 src/cma.c               |    2 ++
 4 files changed, 13 insertions(+), 1 deletion(-)

Index: librdmacm/examples/rping.c
===================================================================
--- librdmacm.orig/examples/rping.c
+++ librdmacm/examples/rping.c
@@ -226,6 +226,10 @@ static int rping_cma_event_handler(struc
 		ret = -1;
 		break;

+	case RDMA_CM_EVENT_ADDR_CHANGE:
+		fprintf(stderr, "ADDR CHANGE event, ignoring\n");
+		break;
+
 	default:
 		fprintf(stderr, "oof bad type!\n");
 		sem_post(&cb->sem);
Index: librdmacm/include/rdma/rdma_cma.h
===================================================================
--- librdmacm.orig/include/rdma/rdma_cma.h
+++ librdmacm/include/rdma/rdma_cma.h
@@ -57,7 +57,8 @@ enum rdma_cm_event_type {
 	RDMA_CM_EVENT_DISCONNECTED,
 	RDMA_CM_EVENT_DEVICE_REMOVAL,
 	RDMA_CM_EVENT_MULTICAST_JOIN,
-	RDMA_CM_EVENT_MULTICAST_ERROR
+	RDMA_CM_EVENT_MULTICAST_ERROR,
+	RDMA_CM_EVENT_ADDR_CHANGE
 };

 enum rdma_port_space {
Index: librdmacm/src/cma.c
===================================================================
--- librdmacm.orig/src/cma.c
+++ librdmacm/src/cma.c
@@ -1438,6 +1438,8 @@ const char *rdma_event_str(enum rdma_cm_
 		return "RDMA_CM_EVENT_MULTICAST_JOIN";
 	case RDMA_CM_EVENT_MULTICAST_ERROR:
 		return "RDMA_CM_EVENT_MULTICAST_ERROR";
+	case RDMA_CM_EVENT_ADDR_CHANGE:
+		return "RDMA_CM_EVENT_ADDR_CHANGE";
 	default:
 		return "UNKNOWN EVENT";
 	}
Index: librdmacm/man/rdma_get_cm_event.3
===================================================================
--- librdmacm.orig/man/rdma_get_cm_event.3
+++ librdmacm/man/rdma_get_cm_event.3
@@ -140,6 +140,11 @@ The multicast join operation (rdma_join_
 An error either occurred joining a multicast group, or, if the group had
 already been joined, on an existing group.  The specified multicast group is
 no longer accessible and should be rejoined, if desired.
+.IP RDMA_CM_EVENT_ADDR_CHANGE_
+The network device associated with this ID through address resolution changed its
+HW address, eg following of bonding failover. This event can serve as a hint for
+applications who want the links used for their RDMA sessions to align with the
+network stack.
 .SH "SEE ALSO"
 rdma_ack_cm_event(3), rdma_create_event_channel(3), rdma_resolve_addr(3),
 rdma_resolve_route(3), rdma_connect(3), rdma_listen(3), rdma_join_multicast(3),