[openib-general] vapi versus openib imm_data

[Roland Dreier]

    Pete> I have an application (PVFS2) that can use either VAPI
    Pete> (ibgd-1.8.2) or OpenIB (libibverbs-1.0.3.1.fc4 and
    Pete> libmthca-1.0.1.fc.4).  Things work just fine between one
    Pete> machine using VAPI and another using OpenIB, but immediate
    Pete> data in an RDMA write comes through byte-swapped.  Both ends
    Pete> are x86_64 hosts.

    Pete> I'm using a heuristic now, knowing the range of values that
    Pete> are valid, but is there any good way to tell if the other
    Pete> side has put its imm_data in network byte order or not?

libmthca (ie openib) puts the immediate data from the work request
directly onto the wire without doing any swapping.  From looking at
the code, it seems that the old VAPI code will byte swap the immediate
data before sending it.

So the simplest solution seems to be for you to use htonl/ntohl when
running on top of the new verbs, and leave it out when running on top
of VAPI.

 - R.