possible rkey byteswap (was Re: [ofw] what's up with the OFA Windows project?)

[frank zago]

Hello Fab,

Fab Tillier wrote:
> Hi Frank,
>
> The fact that the Windows and Linux stacks treat the rkey differently in the SW should not affect wire interop.  In the Windows stack, any values that end up on the wire (like the RKEY in the RDMA header) are always expressed in network order.  They're tokens anyhow and swapping them isn't optimal - you end up with the HCA driver swapping it when returning it to the user, just so the user can swap it to put it on the wire, so that the recipient can swap it back to host order, just to have the HCA driver swap it back to network order when performing the RDMA operation.
>
> So on the Linux side, applications have to swap the rkey and treat it as a host-order value.  On the Windows side they don't.  Can you elaborate on what prevents clean RDMA interop between Linux and Windows?
>
> -Fab
>   
I have an application that runs on windows and linux. The code is the 
same. If I do linux<->linux, or windows<->windows, it works. However 
when I do linux<->windows, I have to swap the rkey for rdma operations.  
To me it seems that either linux or windows is byteswapping the rkey 
that's returned to userspace. The current fix is for my app to byteswap 
the rkey when the memory is registered and byteswap it again when 
posting the rdma work request, but only on one of the OS.

Unfortunateley I didn't have the time to fully investigate with an cable 
analyzer to see who is misbehaving or if my analysis is actually correct.

Frank.