possible rkey byteswap (was Re: [ofw] what's up with the OFA Windows project?)

[Fab Tillier]

Right, the same code won't work properly.  You have to swap on one of the machines, it doesn't really matter which unless you care about how the data is on the wire.  If you care about what it looks like on the wire, you should swap on the Linux side so that the RKey is always in network order when going over the network:

- If the Windows machine is exposing a buffer for RDMA access by a Linux machine, send the RKey as returned by IBAL (it's already in network order).  On the Linux machine, swap the received RKey into host order to use it in an RDMA operation.  The Linux HCA driver will swap the RKey back to network order when putting it on the wire (in the RDMA header).

- If the Linux machine is exposing a buffer for RDMA access by a Windows machine, swap the RKey returned by the OFED stack into network byte order before sending it to the other application.  On the Windows machine, use the RKey value as provided.  The HCA driver won't swap it, and put it on the wire as provided (in the RDMA header).

In IBAL, values that are in network order are identified by their type (net32 or ib_net32, vs. uint32 for host order).  There's no type safety, but the API declarations document it.

Cheers,
-Fab

-----Original Message-----
From: frank zago [mailto:fzago at systemfabricworks.com]
Sent: Wednesday, February 06, 2008 11:31 AM
To: Fab Tillier
Cc: ofw at lists.openfabrics.org
Subject: possible rkey byteswap (was Re: [ofw] what's up with the OFA Windows project?)

Hello Fab,

Fab Tillier wrote:
> Hi Frank,
>
> The fact that the Windows and Linux stacks treat the rkey differently in the SW should not affect wire interop.  In the Windows stack, any values that end up on the wire (like the RKEY in the RDMA header) are always expressed in network order.  They're tokens anyhow and swapping them isn't optimal - you end up with the HCA driver swapping it when returning it to the user, just so the user can swap it to put it on the wire, so that the recipient can swap it back to host order, just to have the HCA driver swap it back to network order when performing the RDMA operation.
>
> So on the Linux side, applications have to swap the rkey and treat it as a host-order value.  On the Windows side they don't.  Can you elaborate on what prevents clean RDMA interop between Linux and Windows?
>
> -Fab
>
I have an application that runs on windows and linux. The code is the
same. If I do linux<->linux, or windows<->windows, it works. However
when I do linux<->windows, I have to swap the rkey for rdma operations.
To me it seems that either linux or windows is byteswapping the rkey
that's returned to userspace. The current fix is for my app to byteswap
the rkey when the memory is registered and byteswap it again when
posting the rdma work request, but only on one of the OS.

Unfortunateley I didn't have the time to fully investigate with an cable
analyzer to see who is misbehaving or if my analysis is actually correct.

Frank.