[openib-general] Re: ANNOUNCE: First usable version of userspace verbs

[Michael S. Tsirkin]

Quoting r. Roland Dreier <roland at topspin.com>:
> Subject: Re: ANNOUNCE: First usable version of userspace verbs
> 
>     Michael> Interestingly, if I rebuild libmthca with -O3 compiler
>     Michael> flag, the pingpong program does not make progress.
>     Michael> Building libibverbs or the test itself with -O3 has no
>     Michael> such effect.
> 
> I can't reproduce this on either i386 or x86-64 (Intel Nocona system).
> 
>     $ gcc --version
>     gcc (GCC) 3.3.5 (Debian 1:3.3.5-8)
>     Copyright (C) 2003 Free Software Foundation, Inc.
>     This is free software; see the source for copying conditions.  There is NO
>     warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
> 
>  - R.
> 

I think I have discovered the problem. It seems that with -O3 my
compiler may reorder the WQE (and possibly CQE) write with respect to
the doorbell.
This wont happen on i386 with consistent i/o ordering since the doorbell
is done in assembly, and probably not on other 32 bit architectures
since the mutex is likely to include a memory barrier.

Applying the folowing patch fixes the problem for me for x86_64.

Signed-off-by: Michael S. Tsirkin <mst at mellanox.co.il>

Index: src/userspace/libmthca/src/doorbell.h
===================================================================
--- src/userspace/libmthca/src/doorbell.h	(revision 1972)
+++ src/userspace/libmthca/src/doorbell.h	(working copy)
@@ -56,6 +56,9 @@ static inline void mthca_write64(uint32_
 
 static inline void mthca_write64(uint32_t val[2], struct mthca_context *ctx, int offset)
 {
+	/* Sufficient for x86_64.
+	 * Other architectures may need a memory barrier here. */
+	asm volatile("" ::: "memory");
 	*(volatile uint64_t *) (ctx->uar + offset) = *(uint64_t *) val;
 }
 

-- 
MST - Michael S. Tsirkin