possible rkey byteswap (was Re: [ofw] what's up with the OFA Windows project?)

[Tom Tucker]

Fab:

This seems broken to me. The app should not have to know whether the remote
peer Linux or Windows.

Tom


On 2/6/08 3:11 PM, "Fab Tillier" <ftillier at windows.microsoft.com> wrote:

> Right, the same code won't work properly.  You have to swap on one of the
> machines, it doesn't really matter which unless you care about how the data is
> on the wire.  If you care about what it looks like on the wire, you should
> swap on the Linux side so that the RKey is always in network order when going
> over the network:
> 
> - If the Windows machine is exposing a buffer for RDMA access by a Linux
> machine, send the RKey as returned by IBAL (it's already in network order).
> On the Linux machine, swap the received RKey into host order to use it in an
> RDMA operation.  The Linux HCA driver will swap the RKey back to network order
> when putting it on the wire (in the RDMA header).
> 
> - If the Linux machine is exposing a buffer for RDMA access by a Windows
> machine, swap the RKey returned by the OFED stack into network byte order
> before sending it to the other application.  On the Windows machine, use the
> RKey value as provided.  The HCA driver won't swap it, and put it on the wire
> as provided (in the RDMA header).
> 
> In IBAL, values that are in network order are identified by their type (net32
> or ib_net32, vs. uint32 for host order).  There's no type safety, but the API
> declarations document it.
> 
> Cheers,
> -Fab
> 
> -----Original Message-----
> From: frank zago [mailto:fzago at systemfabricworks.com]
> Sent: Wednesday, February 06, 2008 11:31 AM
> To: Fab Tillier
> Cc: ofw at lists.openfabrics.org
> Subject: possible rkey byteswap (was Re: [ofw] what's up with the OFA Windows
> project?)
> 
> Hello Fab,
> 
> Fab Tillier wrote:
>> Hi Frank,
>> 
>> The fact that the Windows and Linux stacks treat the rkey differently in the
>> SW should not affect wire interop.  In the Windows stack, any values that end
>> up on the wire (like the RKEY in the RDMA header) are always expressed in
>> network order.  They're tokens anyhow and swapping them isn't optimal - you
>> end up with the HCA driver swapping it when returning it to the user, just so
>> the user can swap it to put it on the wire, so that the recipient can swap it
>> back to host order, just to have the HCA driver swap it back to network order
>> when performing the RDMA operation.
>> 
>> So on the Linux side, applications have to swap the rkey and treat it as a
>> host-order value.  On the Windows side they don't.  Can you elaborate on what
>> prevents clean RDMA interop between Linux and Windows?
>> 
>> -Fab
>> 
> I have an application that runs on windows and linux. The code is the
> same. If I do linux<->linux, or windows<->windows, it works. However
> when I do linux<->windows, I have to swap the rkey for rdma operations.
> To me it seems that either linux or windows is byteswapping the rkey
> that's returned to userspace. The current fix is for my app to byteswap
> the rkey when the memory is registered and byteswap it again when
> posting the rdma work request, but only on one of the OS.
> 
> Unfortunateley I didn't have the time to fully investigate with an cable
> analyzer to see who is misbehaving or if my analysis is actually correct.
> 
> Frank.
> 
> _______________________________________________
> ofw mailing list
> ofw at lists.openfabrics.org
> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ofw