[openib-general] cma private data length

[Caitlin Bestler]

openib-general-bounces at openib.org wrote:
> On Fri, 2006-05-12 at 12:03, Steve Wise wrote:
>> Sean/IB experts:
>> 
>> I'm running a version of rdma_bw from src/userspace/perftest that I
>> ported to utilize the RDMA CMA library for connection setup (stay
>> tuned for a patch to offer this to the trunk).  The CMA version
>> exchanges the rkey information during RDMA connection setup using
>> private data.  I noticed that over IB the private data length
>> received is != to the size of the private data submitted by the
>> application. I send 24 bytes both ways and always get 56 bytes in the
>> RDMA_CM_EVENT_CONNECT_REQUEST event and 196 bytes on the
> RDMA_CM_EVENT_ESTABLISHED event.
> 
> That's what I'd expect as a full MAD lengthed packets are
> being exchanged and the private data length is not encoded in
> the MAD. It would need to be an explicit field in the
> consumer private data.
> 
> CM REQ private data is 736 bits which is 92 bytes, REP private data is
> 1568 bits (196 bytes). Some of the REQ private data is used
> for CMA and the consumer private data is 56 bytes. See IP
> addressing annex.
> 
>> The first 24 bytes, however, are correct so the data is being
>> exchanged.  Over iWARP, the lengths are 24...
> 
>> Is this a bug in the CMA or IB CM?
> 
> Neither IMO.
> 

There are caveats in DAT about this. They should be cut and paste
into the CMA doc as well. Basically, the transport MAY round up the
size of the private data to transport-specific boundaries.

The alternative is to *require* a length field, which would 
decrease the length available for data over transports that
did not encode the length already. So it is clearly better
to round up, and let applications that truly require a
variable length private data message to encode their own
length.

Then the only thing to decide is whether the extra bytes
are undefined or zeroes. Have the sender zero out the
full buffer is pretty cheap compared the establishing
a connection, so zero padding makes sense to me.