[ofa-general] Multithreaded iWARP application

[Philip Frey]

Hello,

I have a peer-to-peer like application where on each peer there is 
thread listening for connection requests. The peers can at the same time 
also actively connect to other peers.

How can the concept or the "rdma_event_channel" now applied to this 
scenario?

Until now I only had one "rdma_event_channel" but with the thread, this 
leads to a race condition where the thread waits for a 
"RDMA_CM_EVENT_CONNECT_REQUST" while the peer tries to actively open a 
connection and is awaiting "RDMA_CM_EVENT_ADDR_RESOLVED" etc.

One solution would be to create a new "rdma_event_channel" for each 
active connection. But what happens at the accepting side? On the 
accepting "rdma_event_channel" (which is now exclusively used for that 
purpose), I get a new "rdma_cm_id" for the connection request from the 
respective event.

Is it now possible to create again a new "rdma_event_channel" for that 
new "rdma_cm_id"? If not, where does the "RDMA_CM_EVENT_ESTABLISHED" 
event go to? (The questionable line is marked with "<--HERE???" in the 
pseudo code below.)


In pseudo code:

/** connecting part **/
struct rdma_cm_id         *id;
struct rdma_event_channel *channel;
struct rdma_cm_event      *event;

channel = rdma_create_event_channel();
rdma_create_id(channel, &id, context, RDMA_PS_TCP);

rdma_resolve_addr(id, src_addr, dst_addr, timeout);
rdma_get_cm_event(channel, &event);	//expecting ADDR_RESOLVED
rdma_ack_cm_event(event);
//same for rdma_resolve_route()		//expecting ROUTE_RESOLVED

rdma_connect(id, conn_param);
rdma_get_event(channel, &event);	//expecting ESTABLISHED
rdma_ack_event(event);

... do RDMA here ...
//disconnect


/** accepting thread **/
struct rdma_cm_id         *listen_id, *id;
struct rdma_event_channel *listen_channel;
struct rdma_cm_event      *listen_event, *event;

channel = rdma_create_event_channel();
rdma_create_id(channel, &id, context, RDMA_PS_TCP);

rdma_bind_addr(id, addr);
rdma_listen(id, backlog);

while(1) {
	rdma_get_cm_event(listen_channel, listen_event);
	rdma_ack_cm_event(listen_event);
	//expecting CONNECT_REQUEST
	id = listen_event->id;
	id->channel = rdma_create_event_channel();	<-- HERE ???
	rdma_accept(id, conn_param);
	rdma_get_cm_event(id->channel, event);
	//expecting ESTABLISHED
	rdma_ack_event(event);

	... do RDMA here ...
	//await disconnect
}


Many thanks for your advice and kind regards!
  Philip