possible rkey byteswap (was Re: [ofw] what's up with the OFA Windows project?)

[Fab Tillier]

Actually, we're in full agreement:

The RKey should always be on the wire in the same, consistent way.  Because the RKey is in the RETH, and is in network order, it makes the most sense to exchange RKeys in network order too.  The difference is that the driver software on Windows and Linux treat the RKey differently.  In Windows it's treated as an opaque value - there's no meaning to swapping it.  In Linux however the implementation expects the RKey to be exchanged in host order across the ABI.  So the Linux software must explicitly swap the RKey it receives from a remote peer, or that it sends to a remote peer, regardless of what OS that remote peer is running.

So the app doesn't need to know what the remote peer is, it just needs to do different things on Windows and Linux to put the same data on the wire.

Make sense?
-Fab

-----Original Message-----
From: ofw-bounces at lists.openfabrics.org [mailto:ofw-bounces at lists.openfabrics.org] On Behalf Of Tom Tucker
Sent: Wednesday, February 06, 2008 1:22 PM
To: Fab Tillier; frank zago
Cc: ofw at lists.openfabrics.org
Subject: Re: possible rkey byteswap (was Re: [ofw] what's up with the OFA Windows project?)

Fab:

This seems broken to me. The app should not have to know whether the remote
peer Linux or Windows.

Tom


On 2/6/08 3:11 PM, "Fab Tillier" <ftillier at windows.microsoft.com> wrote:

> Right, the same code won't work properly.  You have to swap on one of the
> machines, it doesn't really matter which unless you care about how the data is
> on the wire.  If you care about what it looks like on the wire, you should
> swap on the Linux side so that the RKey is always in network order when going
> over the network:
>
> - If the Windows machine is exposing a buffer for RDMA access by a Linux
> machine, send the RKey as returned by IBAL (it's already in network order).
> On the Linux machine, swap the received RKey into host order to use it in an
> RDMA operation.  The Linux HCA driver will swap the RKey back to network order
> when putting it on the wire (in the RDMA header).
>
> - If the Linux machine is exposing a buffer for RDMA access by a Windows
> machine, swap the RKey returned by the OFED stack into network byte order
> before sending it to the other application.  On the Windows machine, use the
> RKey value as provided.  The HCA driver won't swap it, and put it on the wire
> as provided (in the RDMA header).
>
> In IBAL, values that are in network order are identified by their type (net32
> or ib_net32, vs. uint32 for host order).  There's no type safety, but the API
> declarations document it.
>
> Cheers,
> -Fab
>
> -----Original Message-----
> From: frank zago [mailto:fzago at systemfabricworks.com]
> Sent: Wednesday, February 06, 2008 11:31 AM
> To: Fab Tillier
> Cc: ofw at lists.openfabrics.org
> Subject: possible rkey byteswap (was Re: [ofw] what's up with the OFA Windows
> project?)
>
> Hello Fab,
>
> Fab Tillier wrote:
>> Hi Frank,
>>
>> The fact that the Windows and Linux stacks treat the rkey differently in the
>> SW should not affect wire interop.  In the Windows stack, any values that end
>> up on the wire (like the RKEY in the RDMA header) are always expressed in
>> network order.  They're tokens anyhow and swapping them isn't optimal - you
>> end up with the HCA driver swapping it when returning it to the user, just so
>> the user can swap it to put it on the wire, so that the recipient can swap it
>> back to host order, just to have the HCA driver swap it back to network order
>> when performing the RDMA operation.
>>
>> So on the Linux side, applications have to swap the rkey and treat it as a
>> host-order value.  On the Windows side they don't.  Can you elaborate on what
>> prevents clean RDMA interop between Linux and Windows?
>>
>> -Fab
>>
> I have an application that runs on windows and linux. The code is the
> same. If I do linux<->linux, or windows<->windows, it works. However
> when I do linux<->windows, I have to swap the rkey for rdma operations.
> To me it seems that either linux or windows is byteswapping the rkey
> that's returned to userspace. The current fix is for my app to byteswap
> the rkey when the memory is registered and byteswap it again when
> posting the rdma work request, but only on one of the OS.
>
> Unfortunateley I didn't have the time to fully investigate with an cable
> analyzer to see who is misbehaving or if my analysis is actually correct.
>
> Frank.
>
> _______________________________________________
> ofw mailing list
> ofw at lists.openfabrics.org
> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ofw


_______________________________________________
ofw mailing list
ofw at lists.openfabrics.org
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ofw