[Openib-windows] wsd over mt23108 data corruption issue (x64)

[Guy Corem]

I'll try to explain why I've suspected memory pin down problem.

On buffers bigger than 32MB, The IoAllocateMdl call in register_segment
(mosal_iobuf.c) fails.

MOSAL_iobuf_register try to call register_segment again with half the
segment size.

When trying to call with buffers >= 64MB, there are at least two
failures of IoAllocateMdl.

Regarding the timing issues:

Can you suggest a way to test it ? (for example where to put
KeStallExecutionProcessor)

It looks too much systematic to me to be a timing issue (the mismatch is
always in the same place). 

Also: If it's timing issue, how do you explain the strict 64MB
threshold?

The problem occurs with different flavors of HCAs.

Guy


-----Original Message-----
From: ftillier.sst at gmail.com [mailto:ftillier.sst at gmail.com] On Behalf
Of Fabian Tillier
Sent: Tuesday, May 09, 2006 7:47 PM
To: Guy Corem
Cc: openib-windows at openib.org; Erez Haba
Subject: Re: [Openib-windows] wsd over mt23108 data corruption issue
(x64)

Hi Guy,

On 5/9/06, Guy Corem <guyc at voltaire.com> wrote:
>
> Hi Fab and Leonid,
>
> While testing an MPI utility that uses large buffers (>= 64MB), I've
> encountered a data corruption bug.
>
> I was able to reproduce it directly over WSD.
>
> Simplest reproducing scenario:
>
> Use pcattcp from
> http://www.pcausa.com/Utilities/ttcpdown1.htm (compiled to
> 32 bit or 64 bit)
>
> Use large file (>= 64MB)
>
> I've used a 90MB text file
>
> Receiver command line: pcattcp.exe -r -s > file2
>
> Sender command line: pcattcp.exe -s -l 150000000 -n 1 -t receiver_ip <
file
>
> When comparing both files, I see that file2 has 512 bytes misplace
after
> about 45MB (512 bytes were sent twice)
>
> The problem doesn't occur on 32 bit machines.
>
> The problem doesn't occur with new mthca low level driver.
>
> I suspect a memory registration problem, but wasn't able to tack it
down,
> yet.

Memory registration works on full-page (4KB) granularity, so a
sub-page mixup is unlikely to be registration related.  If you saw 4KB
repeated or put in the wrong place, then I'd agree with you.

This sounds like a timing issue - the timing on 32-bit is different
than on 64-bit.  Likewise, the timings of the two HCA drivers are
different.  I suspect that 32-bit and MTHCA runs are just masking the
problem, and that it's not actually an issue with the HCA driver.

> Did you encounter similar problems in the past with huge buffers?

I have not personally seen this issue, but it may be due to my systems
have different timings such that the problem doesn't occur.

Erez Haba is working on a similar issue, but in his case it looks like
a problem in the WSD switch code, and with transfers smaller than
64MB.

- Fab