[openib-general] Re: PATCH] enhancement to rdma_bw and rdma_lat to utilize the RDMA CM

[Steve Wise]

On Tue, 2006-05-16 at 17:41 +0300, Michael S. Tsirkin wrote:
> Quoting r. Steve Wise <swise at opengridcomputing.com>:
> > Subject: PATCH] enhancement to rdma_bw and rdma_lat to utilize the RDMA CM
> > 
> > I don't know who maintains src/userspace/perftest, but here is a patch
> > set that enables rdma_bw and rdma_lat to use the RDMA_CM with the
> > addition of the -c or --cma flag.
> > 
> 
> I'm worried that this makes the program too big. Maybe this should be
> another test rather than an option?
> 

ok.  You want it as a separate pair of programs?

> > The rkey/addr info is exchanged in the private data, and SEND/RECV's are used
> > to sync the client/server before and after execution.
> 
> Do we really need SEND/RECV messages for this?
> I think I get completion with error once the remote side has disconnected. No?
> 

perhaps.  I just thought it was cleaner to synch up at the end.  Just
like the non-cma version does over the TCP socket (see
pp_client_exch_dest() / pp_server_exch_dest() at the end of the test).

> > Also, I added -P or --poll to rdma_bw to allow blocking for completion
> > events when none are ready (if you omit -P, it will block when no
> > completion is available, otherwise it will spin).
> 
> Needs to be a separate patch.

ok.


> 
> > Signed-off-by: Steve Wise <swise at opengridcomputing.com>
> 
> 
> > Index: rdma_lat.c
> > ===================================================================
> > --- rdma_lat.c	(revision 7050)
> > +++ rdma_lat.c	(working copy)
> > @@ -53,6 +53,7 @@
> >  #include <time.h>
> >  
> >  #include <infiniband/verbs.h>
> > +#include <rdma/rdma_cma.h>
> >  
> >  #include "get_clock.h"
> >  
> > @@ -71,7 +72,8 @@
> >  	struct ibv_context *context;
> >  	struct ibv_pd      *pd;
> >  	struct ibv_mr      *mr;
> > -	struct ibv_cq      *cq;
> > +	struct ibv_cq      *scq;
> > +	struct ibv_cq      *rcq;
> 
> Why are you adding another CQ?
> 

It makes waiting for a recv completion easier since you won't get a send
completion when the CQ is only for receives...


> >  	struct ibv_qp      *qp;
> >  	void               *buf;
> >  	volatile char      *post_buf;
> > @@ -80,6 +82,7 @@
> >  	int                 tx_depth;
> >  	struct ibv_sge list;
> >  	struct ibv_send_wr wr;
> > +	struct rdma_cm_id  *cm_id;
> >  };
> >  
> >  struct pingpong_dest {
> > @@ -323,16 +326,22 @@
> >  		return NULL;
> >  	}
> >  
> > -	ctx->cq = ibv_create_cq(ctx->context, tx_depth, NULL, NULL, 0);
> > -	if (!ctx->cq) {
> > +	ctx->rcq = ibv_create_cq(ctx->context, 1, NULL, NULL, 0);
> > +	if (!ctx->rcq) {
> >  		fprintf(stderr, "Couldn't create CQ\n");
> >  		return NULL;
> >  	}
> 
> CQ of depth 1?
> 

Yes, there is only ever one outstanding send/recv exchange...