[openib-general] Re: diff-perftest-07 replace pp_get_local_lid()

[Grant Grundler]

On Wed, May 18, 2005 at 04:51:30PM +0300, Michael S. Tsirkin wrote:
> Quoting r. Grant Grundler <iod00d at hp.com>:
> > Following patch to rdma_lat.c:
> > o replaces pp_get_local_lid with code from ibv_pingpong.
> >   This calls into libibverbs instead of fishing around in /sys FS.
> > 
> > o makes two minor white space fix-ups.
> > 
> > Signed-off-by: Grant Grundler <iod00d at hp.com>
> 
> No patch was attached :(

Doh! sorry...appended now

> > I'd like to slowly restructure main() into multiple distinct parts:
> > 	1) parameter parsing/setting
> > 	2) global data init (e.g. srand())
> > 	3) setup connection
> > 	3) negotiate test+parameters with server/client
> > 	4) run test (maybe several iterations with different params)
> > 	5) exit/cleanup
> 
> Good, but lets pass data around in function parameters
> instead of in global data section.

Certainly. There should be very little/no global data
when I'm done. srand/errno type stuff should be about it.

> > I'm thinking about how to keep the server running and iterating.
> > The goal is to be able to run a sequence of tests just
> > from the client side.
> 
> I just do ssh server $PWD/rdma_lat & ( sleep 1 && $PWD/rdma_lat)

hrm...that would work too though it's not as intuitive.

> > Or is this a waste of time?
> > Should I rather be looking at fixing up netperf to support IB?
> 
> That may be kind of hard, given that uverbs API is completely different
> from socket API.

Agreed - but that's not the part I'm trying to leverage.
I want:
o statistical infrastructure.
o automatically run multiple iterations until confidence level is reached
o CPU utilization (aka "Service Demand")
o bind task to a particular CPU

I can add those things to rmda_lat but it seems like it's a wash
to add IB support to netperf. And since netperf.org is pretty well
known and no-one has a clue about rdma_lat..it seems like a more
obvious place to do the work. I guess that's a political
argument for doing the work in netperf. I'll think about it more
and talk to the netperf maintainer later this week.

thanks,
grant

Signed-off-by: Grant Grundler <iod00d at hp.com>

--- rdma_lat.c-06	2005-05-17 08:50:35.000000000 -0700
+++ rdma_lat.c	2005-05-17 13:19:56.000000000 -0700
@@ -91,31 +91,17 @@
 	unsigned long long vaddr;
 };
 
-/*
- * pp_get_local_lid() uses a pretty bogus method for finding the LID
- * of a local port.  Please don't copy this into your app (or if you
- * do, please rip it out soon).
- */
-static uint16_t pp_get_local_lid(struct ibv_device *dev, int port)
-{
-	char path[256];
-	char val[16];
-	char *name;
 
-	if (sysfs_get_mnt_path(path, sizeof path)) {
-		fprintf(stderr, "Couldn't find sysfs mount.\n");
-		return 0;
-	}
+/* shamelessly taken from src/userspace/libibverbs/examples/pingpong.c */
 
-	asprintf(&name, "%s/class/infiniband/%s/ports/%d/lid", path,
-		 ibv_get_device_name(dev), port);
+static uint16_t pp_get_local_lid(struct pingpong_context *ctx, int port)
+{
+	struct ibv_port_attr attr;
 
-	if (sysfs_read_attribute_value(name, val, sizeof val)) {
-		fprintf(stderr, "Couldn't read LID at %s\n", name);
+	if (ibv_query_port(ctx->context, port, &attr))
 		return 0;
-	}
 
-	return strtol(val, NULL, 0);
+	return attr.lid;
 }
 
 static int pp_client_connect(const char *servername, int port)
@@ -698,7 +684,7 @@
 	/* Create connection between client and server.
 	 * We do it by exchanging data over a TCP socket connection. */
 
-	my_dest.lid = pp_get_local_lid(ib_dev, ib_port);
+	my_dest.lid = pp_get_local_lid(ctx, ib_port);
 	my_dest.qpn = ctx->qp->qp_num;
 	my_dest.psn = lrand48() & 0xffffff;
 	if (!my_dest.lid) {
@@ -758,10 +744,7 @@
 	scnt = 0;
 	rcnt = 0;
 	ccnt = 0;
-	if (servername)
-		client_first_post = 1;
-	else
-		client_first_post = 0;
+	client_first_post = (servername != 0);
 
 	poll_buf = ctx->poll_buf;
 	post_buf = ctx->post_buf;
@@ -815,7 +798,7 @@
 			}
 			if (wc.status != IBV_WC_SUCCESS) {
 				fprintf(stderr, "Completion wth error at %s:\n",
-					servername?"client":"server");
+					servername ? "client" : "server");
 				fprintf(stderr, "Failed status %d: wr_id %d\n",
 					wc.status, (int) wc.wr_id);
 				fprintf(stderr, "scnt=%d, rcnt=%d, ccnt=%d\n",