[ofa-general] Multithreaded iWARP application

[Sean Hefty]

>In pseudo code:
>
>/** connecting part **/
>struct rdma_cm_id         *id;
>struct rdma_event_channel *channel;
>struct rdma_cm_event      *event;
>
>channel = rdma_create_event_channel();
>rdma_create_id(channel, &id, context, RDMA_PS_TCP);
>
>rdma_resolve_addr(id, src_addr, dst_addr, timeout);
>rdma_get_cm_event(channel, &event);	//expecting ADDR_RESOLVED
>rdma_ack_cm_event(event);
>//same for rdma_resolve_route()		//expecting ROUTE_RESOLVED
>
>rdma_connect(id, conn_param);
>rdma_get_event(channel, &event);	//expecting ESTABLISHED
>rdma_ack_event(event);
>
>... do RDMA here ...
>//disconnect
>
>
>/** accepting thread **/
>struct rdma_cm_id         *listen_id, *id;
>struct rdma_event_channel *listen_channel;
>struct rdma_cm_event      *listen_event, *event;
>
>channel = rdma_create_event_channel();
>rdma_create_id(channel, &id, context, RDMA_PS_TCP);
>
>rdma_bind_addr(id, addr);
>rdma_listen(id, backlog);
>
>while(1) {
>	rdma_get_cm_event(listen_channel, listen_event);
>	rdma_ack_cm_event(listen_event);
>	//expecting CONNECT_REQUEST
>	id = listen_event->id;
>	id->channel = rdma_create_event_channel();	<-- HERE ???

This is disallowed.  The kernel is still maintaining the association between the
new id and its current channel, so will still deliver events to the old event
channel for that id.  I think the solution that you're looking for is the call
rdma_migrate_id().  The listen request will have its own event channel, and you
can migrate new connection(s) to separate channel(s).  Depending on your app,
you may be able to get away with a total of 2 channels - one for the listen, and
another one for the connected id's.

As a note, rdma_migrate_id() is a relatively new call.  So I don't know if your
installation has it.  Without it, you're stuck using a single event channel on
the listening side.

- Sean