<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META http-equiv=Content-Type content="text/html; charset=us-ascii">
<STYLE></STYLE>
<META content="MSHTML 6.00.6000.16544" name=GENERATOR></HEAD>
<BODY bgColor=#ffffff>
<DIV dir=ltr align=left><SPAN class=750243114-15112007><FONT face=Arial
color=#0000ff size=2>I don't have an available Linux <--> windows setup,
can you please send me the following data:</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=750243114-15112007><FONT face=Arial
color=#0000ff size=2>* LID values (from ibv_devinfo/vstat)</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=750243114-15112007><FONT face=Arial
color=#0000ff size=2>* the values that you set to the AV.LID (the value that you
set, after fixing/handling endianess)</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=750243114-15112007><FONT face=Arial
color=#0000ff size=2>* the QP number in each size</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=750243114-15112007><FONT face=Arial
color=#0000ff size=2>* the remote QP number that you set in each
side</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=750243114-15112007><FONT face=Arial
color=#0000ff size=2></FONT></SPAN> </DIV>
<DIV dir=ltr align=left><SPAN class=750243114-15112007><FONT face=Arial
color=#0000ff size=2></FONT></SPAN> </DIV>
<DIV dir=ltr align=left><SPAN class=750243114-15112007><FONT face=Arial
color=#0000ff size=2>thanks</FONT></SPAN></DIV>
<DIV dir=ltr align=left><FONT face=Arial size=2>Dotan</FONT></DIV>
<DIV> </DIV><BR>
<BLOCKQUOTE dir=ltr
style="PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT: #0000ff 2px solid; MARGIN-RIGHT: 0px">
<DIV class=OutlookMessageHeader lang=en-us dir=ltr align=left>
<HR tabIndex=-1>
<FONT face=Tahoma size=2><B>From:</B> Diego Guella
[mailto:diego.guella@sircomtech.com] <BR><B>Sent:</B> Wednesday, November 14,
2007 12:11 PM<BR><B>To:</B> Dotan Barak; Tzachi Dar; Fab Tillier<BR><B>Cc:</B>
ofw@lists.openfabrics.org<BR><B>Subject:</B> Re: Completion with bad status:
IBV_WC_RETRY_EXC_ERROR<BR></FONT><BR></DIV>
<DIV></DIV>
<DIV><FONT face=Arial size=2>[resend 1 because of attachment
problems]</FONT></DIV>
<DIV><FONT face=Arial size=2></FONT> </DIV>
<DIV><FONT face=Arial size=2></FONT> </DIV>
<DIV><FONT face=Arial size=2>Hi Dotan,<BR> <BR>About the failure in RDMA
Read, do you mean the garbage data that's printed out?<BR>I don't mind about
that for now, that's expected because the buffer isn't initialized in the
client.<BR> <BR> <BR>I already thought about endianness of the LID,
and tried to change endianness but with no success.<BR> <BR>But now I
modified the code in both Windows and Linux to send out the LID in network
order.<BR>The Windows side does nothing(sends and receives the LID as
is).<BR>The Linunx side changes endianness of the LID before sending, and
changes endianness of the received LID.<BR>(Attached new sources and
executables)<BR> <BR> <BR>But the problem is still
there.<BR> <BR> <BR>Do you have any other
ideas?<BR> <BR> <BR>Thanks,<BR>Diego<BR></DIV></FONT>
<BLOCKQUOTE dir=ltr
style="PADDING-RIGHT: 0px; PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT: #000000 2px solid; MARGIN-RIGHT: 0px">
<DIV style="FONT: 10pt arial">----- Original Message ----- </DIV>
<DIV
style="BACKGROUND: #e4e4e4; FONT: 10pt arial; font-color: black"><B>From:</B>
<A title=dotanb@mellanox.co.il href="mailto:dotanb@mellanox.co.il">Dotan
Barak</A> </DIV>
<DIV style="FONT: 10pt arial"><B>To:</B> <A
title=diego.guella@sircomtech.com
href="mailto:diego.guella@sircomtech.com">Diego Guella</A> ; <A
title=tzachid@mellanox.co.il href="mailto:tzachid@mellanox.co.il">Tzachi
Dar</A> ; <A title=ftillier@windows.microsoft.com
href="mailto:ftillier@windows.microsoft.com">Fab Tillier</A> </DIV>
<DIV style="FONT: 10pt arial"><B>Cc:</B> <A title=ofw@lists.openfabrics.org
href="mailto:ofw@lists.openfabrics.org">ofw@lists.openfabrics.org</A> </DIV>
<DIV style="FONT: 10pt arial"><B>Sent:</B> Wednesday, November 14, 2007 9:57
AM</DIV>
<DIV style="FONT: 10pt arial"><B>Subject:</B> RE: Completion with bad
status: IBV_WC_RETRY_EXC_ERROR</DIV>
<DIV><BR></DIV>
<DIV dir=ltr align=left><SPAN class=078235508-14112007><FONT face=Arial
color=#0000ff size=2>Hi.</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=078235508-14112007><FONT face=Arial
color=#0000ff size=2></FONT></SPAN> </DIV>
<DIV dir=ltr align=left><SPAN class=078235508-14112007><FONT face=Arial
color=#0000ff size=2>I checked your code in Linux and you have a failure in
RDMA Read...</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=078235508-14112007><FONT face=Arial
color=#0000ff size=2></FONT></SPAN> </DIV>
<DIV dir=ltr align=left><SPAN class=078235508-14112007><FONT face=Arial
color=#0000ff size=2> But for your problem: in windows the LID of the
port is in network order and in Linux the LID of the port is in host
order.</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=078235508-14112007><FONT face=Arial
color=#0000ff size=2>(order means endianess).</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=078235508-14112007><FONT face=Arial
color=#0000ff size=2></FONT></SPAN> </DIV>
<DIV dir=ltr align=left><SPAN class=078235508-14112007><FONT face=Arial
color=#0000ff size=2></FONT></SPAN> </DIV>
<DIV dir=ltr align=left><SPAN class=078235508-14112007><FONT face=Arial
color=#0000ff size=2>Fixing this should solve the problem
...</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=078235508-14112007><FONT face=Arial
color=#0000ff size=2></FONT></SPAN> </DIV>
<DIV><FONT face=Arial color=#0000ff size=2></FONT> </DIV>
<DIV dir=ltr align=left><FONT face=Arial size=2>Dotan</FONT></DIV>
<DIV> </DIV><BR>
<BLOCKQUOTE dir=ltr
style="PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT: #0000ff 2px solid; MARGIN-RIGHT: 0px">
<DIV class=OutlookMessageHeader lang=en-us dir=ltr align=left>
<HR tabIndex=-1>
<FONT face=Tahoma size=2><B>From:</B> Diego Guella
[mailto:diego.guella@sircomtech.com] <BR><B>Sent:</B> Wednesday, November
14, 2007 10:29 AM<BR><B>To:</B> Dotan Barak; Tzachi Dar; Fab
Tillier<BR><B>Cc:</B> <A
href="mailto:ofw@lists.openfabrics.org">ofw@lists.openfabrics.org</A><BR><B>Subject:</B>
Re: Completion with bad status:
IBV_WC_RETRY_EXC_ERROR<BR></FONT><BR></DIV>
<DIV></DIV>
<DIV><FONT face=Arial size=2>Hi Dotan,</FONT></DIV>
<DIV><FONT face=Arial size=2></FONT> </DIV>
<DIV><FONT face=Arial size=2>I apologize for that silly mistake, obviously
WR_SEND is different from IBV_WR_SEND, and the same was for WR_RDMA_READ,
etc. etc...</FONT></DIV>
<DIV><FONT face=Arial size=2></FONT> </DIV>
<DIV><FONT face=Arial size=2>So, I removed <iba/ib_types.h> from the
includes, to make sure I don't use them.</FONT></DIV>
<DIV><FONT face=Arial size=2></FONT> </DIV>
<DIV><FONT face=Arial size=2>Now the Linux program works with
send/recv and rdma to himself (daemon/client on the same machine), but I
still get the same error when I try communication between
Windows/Linux.</FONT></DIV>
<DIV><FONT face=Arial size=2>The error applies to SEND, RDMA_WRITE,
RDMA_READ, and using the daemon both on Linux or Windows.</FONT></DIV>
<DIV><FONT face=Arial size=2></FONT> </DIV>
<DIV><FONT face=Arial size=2>Attached are the new sources (note that
Windows sources aren't changed).</FONT></DIV>
<DIV><FONT face=Arial size=2></FONT> </DIV>
<DIV><FONT face=Arial size=2></FONT> </DIV>
<DIV><FONT face=Arial size=2></FONT> </DIV>
<DIV><FONT face=Arial size=2>Thanks,</FONT></DIV>
<DIV><FONT face=Arial size=2>Diego</FONT></DIV>
<DIV><FONT face=Arial size=2></FONT> </DIV>
<DIV><FONT face=Arial size=2></FONT> </DIV>
<DIV><FONT face=Arial size=2></FONT> </DIV>
<BLOCKQUOTE dir=ltr
style="PADDING-RIGHT: 0px; PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT: #000000 2px solid; MARGIN-RIGHT: 0px">
<DIV style="FONT: 10pt arial">----- Original Message ----- </DIV>
<DIV
style="BACKGROUND: #e4e4e4; FONT: 10pt arial; font-color: black"><B>From:</B>
<A title=dotanb@mellanox.co.il href="mailto:dotanb@mellanox.co.il">Dotan
Barak</A> </DIV>
<DIV style="FONT: 10pt arial"><B>To:</B> <A
title=diego.guella@sircomtech.com
href="mailto:diego.guella@sircomtech.com">Diego Guella</A> ; <A
title=tzachid@mellanox.co.il href="mailto:tzachid@mellanox.co.il">Tzachi
Dar</A> ; <A title=ftillier@windows.microsoft.com
href="mailto:ftillier@windows.microsoft.com">Fab Tillier</A> </DIV>
<DIV style="FONT: 10pt arial"><B>Cc:</B> <A
title=ofw@lists.openfabrics.org
href="mailto:ofw@lists.openfabrics.org">ofw@lists.openfabrics.org</A>
</DIV>
<DIV style="FONT: 10pt arial"><B>Sent:</B> Tuesday, November 13, 2007
3:13 PM</DIV>
<DIV style="FONT: 10pt arial"><B>Subject:</B> RE: Completion with bad
status: IBV_WC_RETRY_EXC_ERROR</DIV>
<DIV><BR></DIV>
<DIV dir=ltr align=left><SPAN class=625085213-13112007><FONT face=Arial
color=#0000ff size=2>o.k., I passed that and the code compilation stage
....</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=625085213-13112007><FONT face=Arial
color=#0000ff size=2></FONT></SPAN> </DIV>
<DIV dir=ltr align=left><SPAN class=625085213-13112007><FONT face=Arial
color=#0000ff size=2></FONT></SPAN><SPAN class=625085213-13112007><FONT
face=Arial color=#0000ff size=2></FONT></SPAN> </DIV>
<DIV dir=ltr align=left><SPAN class=625085213-13112007><FONT face=Arial
color=#0000ff size=2>Just a small note:</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=625085213-13112007><FONT face=Arial
color=#0000ff size=2>In the modify_qp: qp_access_flags is only for
supported remote operations, so IBV_ACCESS_LOCAL_WRITE should be
removed</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=625085213-13112007><FONT face=Arial
color=#0000ff size=2></FONT></SPAN> </DIV>
<DIV dir=ltr align=left><SPAN class=625085213-13112007><FONT face=Arial
color=#0000ff size=2></FONT></SPAN> </DIV>
<DIV dir=ltr align=left><SPAN class=625085213-13112007><FONT face=Arial
color=#0000ff size=2>I understand what the root cause of the problem is:
you took a code from windows and moved only PART of the code to
Linux.</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=625085213-13112007><FONT face=Arial
color=#0000ff size=2></FONT></SPAN> </DIV>
<DIV dir=ltr align=left><SPAN class=625085213-13112007><FONT face=Arial
color=#0000ff size=2>For example: WR_SEND is defined in iba/ib_types and
has the value 1 (which is RDMA WRITE_WITH_IMM in
Linux)</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=625085213-13112007><FONT face=Arial
color=#0000ff size=2>so actually, you did RDMA write with rkey and
remote address with undefined values.</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=625085213-13112007><FONT face=Arial
color=#0000ff size=2>(this is only an example for the corruption that
happens during the test execution because of this
issue)</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=625085213-13112007><FONT face=Arial
color=#0000ff size=2></FONT></SPAN> </DIV>
<DIV dir=ltr align=left><SPAN class=625085213-13112007><FONT face=Arial
color=#0000ff size=2>The code passed compilation because you included
iba/ib_types.h (in types.h) in Linux too.</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=625085213-13112007><FONT face=Arial
color=#0000ff size=2>This file should not be included in Linux (unless
you really need it ...)</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=625085213-13112007><FONT face=Arial
color=#0000ff size=2></FONT></SPAN> </DIV>
<DIV dir=ltr align=left><SPAN class=625085213-13112007><FONT face=Arial
color=#0000ff size=2>All of the structures, functions, enumerations in
Linux verbs start with IBV_ or ibv_.</FONT></SPAN></DIV>
<DIV dir=ltr align=left><SPAN class=625085213-13112007><FONT face=Arial
color=#0000ff size=2></FONT></SPAN> </DIV>
<DIV dir=ltr align=left><SPAN class=625085213-13112007><FONT face=Arial
color=#0000ff size=2></FONT></SPAN> </DIV>
<DIV dir=ltr align=left><SPAN class=625085213-13112007><FONT face=Arial
color=#0000ff size=2>This should fix the test problems (I hope
...)</FONT></SPAN></DIV>
<DIV><FONT face=Arial color=#0000ff size=2></FONT> </DIV>
<DIV><FONT face=Arial color=#0000ff size=2></FONT> </DIV>
<DIV><SPAN class=625085213-13112007><FONT face=Arial color=#0000ff
size=2>thanks</FONT></SPAN></DIV>
<DIV dir=ltr align=left><FONT face=Arial size=2>Dotan</FONT></DIV>
<DIV><FONT face=Arial
size=2></FONT> </DIV></BLOCKQUOTE></BLOCKQUOTE></BLOCKQUOTE></BLOCKQUOTE></BODY></HTML>