[Openib-windows] [PATCH] MTHCA: always treat RKEY in network order

[Leonid Keller]

I didn't study your patch profoundly, but I have a feeling that it
doesn't fix anything.
I mean, it's true that the driver mistakenly returns rkey and lkey in LE
upon MR creation, but it then mistakenly :) converts them to BE on post
send request. So it is to be fixed carefully, yes, but it seems not to
solve any problem.

I've tried the current code - without your patch - with our test. All
worked OK.

Here is one example:
	qp_test.exe --daemon
	qp_test --ip=10.4.3.27 --qp=5 --oust=100 --thread=4 -i=10000
--grh CLIENT RR 5 10240 SERVER RW 5 10240 CLIENT RWI 5 10240
The latter means:
	The client part creates 4 threads, 5 QPs in each thread and
execute 3 operations, 10000 iterations each:
		RDMA read as client with 5 segments of 10K each;
		RDMA write as server with 5 segments of 10K each;
		RDMA write with immediate as client with 5 segments of
10K each;
	There are 100 outstanding requests on every QP and packets are
sent with GRH header.
	The test worked in polling mode, transport was RC (it also
worked with UC).

It was tried on x64 machine with Artavor (Arbel in Tavor mode).

With what parameters do you use RDMA in WSD ?


> -----Original Message-----
> From: ftillier.sst at gmail.com [mailto:ftillier.sst at gmail.com] 
> On Behalf Of Fabian Tillier
> Sent: Sunday, April 02, 2006 12:59 PM
> To: Leonid Keller
> Cc: openib-windows at openib.org
> Subject: Re: [Openib-windows] [PATCH] MTHCA: always treat 
> RKEY in network order
> 
> On 4/2/06, Leonid Keller <leonid at mellanox.co.il> wrote:
> > About the patch: if changing, I think, we need to change 
> both rkey and 
> > lkey to be consistent.
> 
> Ok, I'll fix it up and resend.  I hadn't changed the lkey 
> since it's only used locally, so it doesn't really matter.
> 
> For the rkey, I've always felt that since it goes on the 
> wire, and really is an opaque value returned by the HCA 
> driver, it was really unnatural for a ULP to byteswap this 
> value.  Plus, it had the added benefit of removing the swap 
> from the WQE formatting, which should make it a little more efficient.
> 
> > About the below stuff: did you try using the new driver on 
> both sides ?
> 
> I did, and it failed too, except that in this case, both RDMA 
> write and RDMA read failed.  In both cases, the error was 
> remote invalid request.
> 
> I was going to put together an RDMA test to see if it's 
> something within the WSD provider, but I doubt it is.  Do you 
> have an RDMA test handy to verify that RDMAs work?
> 
> Looking at the MTT and MPT dumps, everything on the MR side 
> of things looks good.  Perhaps the QP is not being put in the 
> proper state? 
> What would cause that specific error (vs some other error)?
> 
> Thanks,
> 
> - Fab
>