possible rkey byteswap (was Re: [ofw] what's up with the OFAWindows project?)

[Robert Pearson]

It is no longer opaque when sent to Linux. No?

-----Original Message-----
From: ofw-bounces at lists.openfabrics.org
[mailto:ofw-bounces at lists.openfabrics.org] On Behalf Of Fab Tillier
Sent: Wednesday, February 06, 2008 3:27 PM
To: Tom Tucker; frank zago
Cc: ofw at lists.openfabrics.org
Subject: RE: possible rkey byteswap (was Re: [ofw] what's up with the
OFAWindows project?)

Actually, we're in full agreement:

The RKey should always be on the wire in the same, consistent way.  Because
the RKey is in the RETH, and is in network order, it makes the most sense to
exchange RKeys in network order too.  The difference is that the driver
software on Windows and Linux treat the RKey differently.  In Windows it's
treated as an opaque value - there's no meaning to swapping it.  In Linux
however the implementation expects the RKey to be exchanged in host order
across the ABI.  So the Linux software must explicitly swap the RKey it
receives from a remote peer, or that it sends to a remote peer, regardless
of what OS that remote peer is running.

So the app doesn't need to know what the remote peer is, it just needs to do
different things on Windows and Linux to put the same data on the wire.

Make sense?
-Fab

-----Original Message-----
From: ofw-bounces at lists.openfabrics.org
[mailto:ofw-bounces at lists.openfabrics.org] On Behalf Of Tom Tucker
Sent: Wednesday, February 06, 2008 1:22 PM
To: Fab Tillier; frank zago
Cc: ofw at lists.openfabrics.org
Subject: Re: possible rkey byteswap (was Re: [ofw] what's up with the OFA
Windows project?)

Fab:

This seems broken to me. The app should not have to know whether the remote
peer Linux or Windows.

Tom


On 2/6/08 3:11 PM, "Fab Tillier" <ftillier at windows.microsoft.com> wrote:

> Right, the same code won't work properly.  You have to swap on one of the
> machines, it doesn't really matter which unless you care about how the
data is
> on the wire.  If you care about what it looks like on the wire, you should
> swap on the Linux side so that the RKey is always in network order when
going
> over the network:
>
> - If the Windows machine is exposing a buffer for RDMA access by a Linux
> machine, send the RKey as returned by IBAL (it's already in network
order).
> On the Linux machine, swap the received RKey into host order to use it in
an
> RDMA operation.  The Linux HCA driver will swap the RKey back to network
order
> when putting it on the wire (in the RDMA header).
>
> - If the Linux machine is exposing a buffer for RDMA access by a Windows
> machine, swap the RKey returned by the OFED stack into network byte order
> before sending it to the other application.  On the Windows machine, use
the
> RKey value as provided.  The HCA driver won't swap it, and put it on the
wire
> as provided (in the RDMA header).
>
> In IBAL, values that are in network order are identified by their type
(net32
> or ib_net32, vs. uint32 for host order).  There's no type safety, but the
API
> declarations document it.
>
> Cheers,
> -Fab
>
> -----Original Message-----
> From: frank zago [mailto:fzago at systemfabricworks.com]
> Sent: Wednesday, February 06, 2008 11:31 AM
> To: Fab Tillier
> Cc: ofw at lists.openfabrics.org
> Subject: possible rkey byteswap (was Re: [ofw] what's up with the OFA
Windows
> project?)
>
> Hello Fab,
>
> Fab Tillier wrote:
>> Hi Frank,
>>
>> The fact that the Windows and Linux stacks treat the rkey differently in
the
>> SW should not affect wire interop.  In the Windows stack, any values that
end
>> up on the wire (like the RKEY in the RDMA header) are always expressed in
>> network order.  They're tokens anyhow and swapping them isn't optimal -
you
>> end up with the HCA driver swapping it when returning it to the user,
just so
>> the user can swap it to put it on the wire, so that the recipient can
swap it
>> back to host order, just to have the HCA driver swap it back to network
order
>> when performing the RDMA operation.
>>
>> So on the Linux side, applications have to swap the rkey and treat it as
a
>> host-order value.  On the Windows side they don't.  Can you elaborate on
what
>> prevents clean RDMA interop between Linux and Windows?
>>
>> -Fab
>>
> I have an application that runs on windows and linux. The code is the
> same. If I do linux<->linux, or windows<->windows, it works. However
> when I do linux<->windows, I have to swap the rkey for rdma operations.
> To me it seems that either linux or windows is byteswapping the rkey
> that's returned to userspace. The current fix is for my app to byteswap
> the rkey when the memory is registered and byteswap it again when
> posting the rdma work request, but only on one of the OS.
>
> Unfortunateley I didn't have the time to fully investigate with an cable
> analyzer to see who is misbehaving or if my analysis is actually correct.
>
> Frank.
>
> _______________________________________________
> ofw mailing list
> ofw at lists.openfabrics.org
> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ofw


_______________________________________________
ofw mailing list
ofw at lists.openfabrics.org
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ofw
_______________________________________________
ofw mailing list
ofw at lists.openfabrics.org
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ofw