possible rkey byteswap (was Re: [ofw] what's up with the OFA Windows project?)

[Hal Rosenstock]

On Wed, 2008-02-06 at 15:22 -0600, Tom Tucker wrote:
> Fab:
> 
> This seems broken to me. The app should not have to know whether the remote
> peer Linux or Windows.

The rule should be what Fab stated in a previous email: big endian
(network order) on the IB "wire". Isn't that sufficient ?

-- Hal

> Tom
> 
> 
> On 2/6/08 3:11 PM, "Fab Tillier" <ftillier at windows.microsoft.com> wrote:
> 
> > Right, the same code won't work properly.  You have to swap on one of the
> > machines, it doesn't really matter which unless you care about how the data is
> > on the wire.  If you care about what it looks like on the wire, you should
> > swap on the Linux side so that the RKey is always in network order when going
> > over the network:
> > 
> > - If the Windows machine is exposing a buffer for RDMA access by a Linux
> > machine, send the RKey as returned by IBAL (it's already in network order).
> > On the Linux machine, swap the received RKey into host order to use it in an
> > RDMA operation.  The Linux HCA driver will swap the RKey back to network order
> > when putting it on the wire (in the RDMA header).
> > 
> > - If the Linux machine is exposing a buffer for RDMA access by a Windows
> > machine, swap the RKey returned by the OFED stack into network byte order
> > before sending it to the other application.  On the Windows machine, use the
> > RKey value as provided.  The HCA driver won't swap it, and put it on the wire
> > as provided (in the RDMA header).
> > 
> > In IBAL, values that are in network order are identified by their type (net32
> > or ib_net32, vs. uint32 for host order).  There's no type safety, but the API
> > declarations document it.
> > 
> > Cheers,
> > -Fab
> > 
> > -----Original Message-----
> > From: frank zago [mailto:fzago at systemfabricworks.com]
> > Sent: Wednesday, February 06, 2008 11:31 AM
> > To: Fab Tillier
> > Cc: ofw at lists.openfabrics.org
> > Subject: possible rkey byteswap (was Re: [ofw] what's up with the OFA Windows
> > project?)
> > 
> > Hello Fab,
> > 
> > Fab Tillier wrote:
> >> Hi Frank,
> >> 
> >> The fact that the Windows and Linux stacks treat the rkey differently in the
> >> SW should not affect wire interop.  In the Windows stack, any values that end
> >> up on the wire (like the RKEY in the RDMA header) are always expressed in
> >> network order.  They're tokens anyhow and swapping them isn't optimal - you
> >> end up with the HCA driver swapping it when returning it to the user, just so
> >> the user can swap it to put it on the wire, so that the recipient can swap it
> >> back to host order, just to have the HCA driver swap it back to network order
> >> when performing the RDMA operation.
> >> 
> >> So on the Linux side, applications have to swap the rkey and treat it as a
> >> host-order value.  On the Windows side they don't.  Can you elaborate on what
> >> prevents clean RDMA interop between Linux and Windows?
> >> 
> >> -Fab
> >> 
> > I have an application that runs on windows and linux. The code is the
> > same. If I do linux<->linux, or windows<->windows, it works. However
> > when I do linux<->windows, I have to swap the rkey for rdma operations.
> > To me it seems that either linux or windows is byteswapping the rkey
> > that's returned to userspace. The current fix is for my app to byteswap
> > the rkey when the memory is registered and byteswap it again when
> > posting the rdma work request, but only on one of the OS.
> > 
> > Unfortunateley I didn't have the time to fully investigate with an cable
> > analyzer to see who is misbehaving or if my analysis is actually correct.
> > 
> > Frank.
> > 
> > _______________________________________________
> > ofw mailing list
> > ofw at lists.openfabrics.org
> > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ofw
> 
> 
> _______________________________________________
> ofw mailing list
> ofw at lists.openfabrics.org
> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ofw