[openib-general] Re: PATCH] enhancement to rdma_bw and rdma_lat to utilize the RDMA CM

[Michael S. Tsirkin]

Quoting r. Steve Wise <swise at opengridcomputing.com>:
> Subject: PATCH] enhancement to rdma_bw and rdma_lat to utilize the RDMA CM
> 
> I don't know who maintains src/userspace/perftest, but here is a patch
> set that enables rdma_bw and rdma_lat to use the RDMA_CM with the
> addition of the -c or --cma flag.
> 

I'm worried that this makes the program too big. Maybe this should be
another test rather than an option?

> The rkey/addr info is exchanged in the private data, and SEND/RECV's are used
> to sync the client/server before and after execution.

Do we really need SEND/RECV messages for this?
I think I get completion with error once the remote side has disconnected. No?

> Also, I added -P or --poll to rdma_bw to allow blocking for completion
> events when none are ready (if you omit -P, it will block when no
> completion is available, otherwise it will spin).

Needs to be a separate patch.

> Signed-off-by: Steve Wise <swise at opengridcomputing.com>


> Index: rdma_lat.c
> ===================================================================
> --- rdma_lat.c	(revision 7050)
> +++ rdma_lat.c	(working copy)
> @@ -53,6 +53,7 @@
>  #include <time.h>
>  
>  #include <infiniband/verbs.h>
> +#include <rdma/rdma_cma.h>
>  
>  #include "get_clock.h"
>  
> @@ -71,7 +72,8 @@
>  	struct ibv_context *context;
>  	struct ibv_pd      *pd;
>  	struct ibv_mr      *mr;
> -	struct ibv_cq      *cq;
> +	struct ibv_cq      *scq;
> +	struct ibv_cq      *rcq;

Why are you adding another CQ?

>  	struct ibv_qp      *qp;
>  	void               *buf;
>  	volatile char      *post_buf;
> @@ -80,6 +82,7 @@
>  	int                 tx_depth;
>  	struct ibv_sge list;
>  	struct ibv_send_wr wr;
> +	struct rdma_cm_id  *cm_id;
>  };
>  
>  struct pingpong_dest {
> @@ -323,16 +326,22 @@
>  		return NULL;
>  	}
>  
> -	ctx->cq = ibv_create_cq(ctx->context, tx_depth, NULL, NULL, 0);
> -	if (!ctx->cq) {
> +	ctx->rcq = ibv_create_cq(ctx->context, 1, NULL, NULL, 0);
> +	if (!ctx->rcq) {
>  		fprintf(stderr, "Couldn't create CQ\n");
>  		return NULL;
>  	}

CQ of depth 1?

-- 
MST