[openib-general] Re: [PATCH] mthca: fix posting lists of 256 entries for tavor

[Michael S. Tsirkin]

Quoting r. Roland Dreier <rdreier at cisco.com>:
> Subject: Re: [PATCH] mthca: fix posting lists of 256 entries for tavor
> 
>     Michael> No idea - the site seems to be down :)
> 
> It's working from here -- must be an issue in your network.
> 
> Anyway the report is:
> 
> *************************************************************
> Host Architecture : x86_64
> Linux Distribution: Fedora Core release 4 (Stentz)
> Kernel Version    : 2.6.11-1.1369_FC4smp
> Memory size       : 4071672 kB
> Driver Version    : OFED-1.0-rc5-pre5
> HCA ID(s)         : mthca0
> HCA model(s)      : 25208
> FW version(s)     : 4.7.600
> Board(s)          : MT_00A0010001
> *************************************************************
> 
> posting a list of multiples of 256 WR to SRQ or QP may be corrupted.
> The WR list that is being posted may be posted to a different QP than
> the QP
> number of the QP handle.
> 
> test to reproduce it: qp_test
> daemon:
> qp_test --daemon
> client:
> qp_test --thread=15 --oust=256 --srq CLIENT SR 1 1
>      or
> qp_test --thread=15 --oust=256 CLIENT SR 1 1
> 

Yep, looks like it.
Testing the following patch now, will let you know.

Index: openib/src/userspace/libmthca/src/srq.c
===================================================================
--- openib/src/userspace/libmthca/src/srq.c	(revision 7317)
+++ openib/src/userspace/libmthca/src/srq.c	(working copy)
@@ -97,24 +97,7 @@
 
 	first_ind = srq->first_free;
 
-	for (nreq = 0; wr; ++nreq, wr = wr->next) {
-		if (nreq == MTHCA_TAVOR_MAX_WQES_PER_RECV_DB) {
-			nreq = 0;
-
-			doorbell[0] = htonl(first_ind << srq->wqe_shift);
-			doorbell[1] = htonl(srq->srqn << 8);
-
-			/*
-			 * Make sure that descriptors are written
-			 * before doorbell is rung.
-			 */
-			mb();
-
-			mthca_write64(doorbell, to_mctx(ibsrq->context), MTHCA_RECV_DOORBELL);
-
-			first_ind = srq->first_free;
-		}
-
+	for (nreq = 0; wr; wr = wr->next) {
 		ind = srq->first_free;
 
 		if (ind < 0) {
@@ -172,6 +155,23 @@
 
 		srq->wrid[ind]  = wr->wr_id;
 		srq->first_free = next_ind;
+
+		if (++nreq == MTHCA_TAVOR_MAX_WQES_PER_RECV_DB) {
+			nreq = 0;
+
+			doorbell[0] = htonl(first_ind << srq->wqe_shift);
+			doorbell[1] = htonl(srq->srqn << 8);
+
+			/*
+			 * Make sure that descriptors are written
+			 * before doorbell is rung.
+			 */
+			mb();
+
+			mthca_write64(doorbell, to_mctx(ibsrq->context), MTHCA_RECV_DOORBELL);
+
+			first_ind = srq->first_free;
+		}
 	}
 
 	if (nreq) {

-- 
MST