[ofa-general] librdmacm feature request

[Doug Ledford]

I've been trying to write some code using librdmacm and I've run across
a few shortcomings in the library.

1)  When you listen for connections, the event includes a new cm_id
struct attached to the listen event channel.  Attempts to change this
channel make the cm_id unusable (rdma_create_qp fails).  This is
suboptimal in situations where you want the listen channel to produce
listen events only.  A function such as rdma_modify_channel(cm_id,
new_channel); would work to solve this.

2)  When you create a new cm_id with the intent of connecting to another
machine, it is again desirable to get your events related to the
establishment of the connection in a separate channel from those events
related to already established connections (amongst other things, if you
are sharing a channel with a different thread that is responsible for
tearing down connections on error, then which thread gets the
ADDR_RESOLVED or ROUTE_RESOLVED events is up in the air...to make sure
it gets delivered properly, I currently have the connecting thread
pthread_mutex_lock the connection construct, set connection->cm_waiting
= 1, then issue the rdma_resolve_route, then pthread_mutex_lock again so
it deadlocks, and then other thread gets the event, checks
connection->cm_waiting == 1, and if true it places the event pointer in
connection->event, clears connection->cm_waiting, then
pthread_mutex_unlock's the connection...how gross is that).  So, using a
separate event channel up until the connection is established, then
calling rdma_modify_channel() would also solve this problem.

3)  The man pages on rdma_connect() and rdma_accept() aren't really
clear on the role of the connection parameters struct that gets passed
in.  Specifically, it doesn't say whether or not the initiator_depth and
responder_resources in the parm struct present in the listen event are
what the other side set, or if they are already swapped to indicate the
minimum/maximum that we can set on our side of the connection.  Also,
the initial message pointer is not detailed.  When we call
rdma_accept/rdma_reject, does our parm struct need to have that same
pointer?  Do we need to free that mem?  Can we supply a new initial
message and not leak the memory associated with the incoming initial
message?

Otherwise I haven't had any problems so far.

-- 
Doug Ledford <dledford at redhat.com>
              GPG KeyID: CFBFF194
              http://people.redhat.com/dledford

Infiniband specific RPMs available at
              http://people.redhat.com/dledford/Infiniband
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20071007/9fb80741/attachment.sig>