[openib-general] Re: RMPP

[Sean Hefty]

Hal Rosenstock wrote:
> Hi Sean,
> 
> In addition to passing the hdr_len and data_len to ib_create_send_mad: 
>                 rmpp_mad->rmpp_hdr.paylen_newwin = cpu_to_be32(hdr_len -
>                         offsetof(struct ib_rmpp_mad, data) + data_len);
> That's not the "real" payload length that would go into the packet; just
> one segment's worth of class header. Correct ?

There are two sizes needed by the RMPP code.  First, it needs the total 
length of the data buffer, which includes any necessary padding.  This is 
set in the scatter-gather entry for the work request.  Second, it needs to 
know how much of the data buffer contains valid user data.

ib_create_send_mad returns the length of the of the valid user data in a 
slightly encoded format.  It subtracts the size of the RMPP and common MAD 
headers from the size (see below).  The intent is that the payload length 
value is set to the correct value for MADs that are a single segment in length.

-------------
MAD header
-------------
RMPP header
-------------
SA/Vendor hdr
-------------
User data
-------------
Pad
-------------

sge length = size from MAD header to end of pad.
payload = size from SA/Vendor hdr to end of user data.

> If the above is correct, I'm not sure the actual payload lengths in the
> DATA packets are correct. I also see the padding on the last segment of
> a multipacket send not cleared (I integrated the part of your patch
> relating to the pad calculation).

I will run some tests and verify the payload values using madeye.  I'm not 
sure why the padding isn't cleared.  It may be an indication that 
create_send_mad isn't allocating the correct pad size.

> In one test case, I see the first RMPP DATA segment sent and there is no
> response (ACK) from the receiver (this was due to a (receiver) test
> program issue). The transmitter retry depends on retries in the send_wr
> ud structure. Does it need to known/Is there a way to know that this
> failed (no ACK, etc.) when retries are exhausted or is this reliant on
> the receiver rerequesting or is the entire RMPP transaction treated like
> a UD send (e.g. unreliable) ?

The RMPP send is treated as reliable, even if no response is expected.  If 
the send is not ACKed completely, the request will timeout, and a send 
failure is reported to the user.  If the send completes successfully, then 
the user knows that it was received by the remote side; although, there's no 
guarantee that it was processed by anyone.

The code will retry a given segment the number of times specified by the 
user.  If forward progress is made on the send, the retry count is reset.

> Here's a summary of changes so far:
> 1. ib_create_send_mad either needs an additional parameter (RMPP active
> in current send packet) or the paylen_newwin needs to be set by the user
> outside of this routine.

I have this on my short term to-do list.  I'm just not sure on the best 
approach yet...

> 2. Some minor ib_mad.h commentary changes for more clarity on the
> assumptions of the RMPP API.