[ofiwg] respecting message boundaries in recv

[Paolo Inaudi]

Hi everybody,
I'm implementing a libfabric provider and I'm unsure about the 
implications of the statement 'Message boundaries are maintained' in 
fi_recv* operations. In particular, what should happen if a receive 
buffer posted with fi_recv (the same applies to fi_recvv or fi_recvmsg) 
is smaller than the size of the message to be received?

I see some options:
1. Nothing is received, and an error is inserted into the completion 
queue (FI_EMSGSIZE seems to be the right error code, with the len field 
containing the message length).
2. Posted buffer is filled, and a completion is generated. If completion 
format is at least FI_CQ_FORMAT_MSG, the len field of the completion is 
set to the size of the message to be received (which means bigger than 
the buffer size; this way the application is informed that the message 
is bigger than expected)
3. Posted buffer is filled, and a completion is generated. If completion 
format is at least FI_CQ_FORMAT_MSG, the len field of the completion is 
set to the size of the buffer. The application must have its own way to 
know that the message is not completely received.
3. Posted buffer is filled, but no completion is generated until a new 
buffer is posted which is big enough to contain the remainder of the 
message (this seems quite bad, since I don't see a practical way for the 
application to know when this happens).

Moreover, given option 2 or 3 is the right one, what should happen when 
another receive buffer is posted?
1. It should be filled with the remainder of the partially read message
2. It should be filled with the partially read message as a whole (i.e. 
the already read part + the remainder)
3. It should be filled with the next message in the queue (that is, the 
remainder gets discarded, and the application must retransmit it)

Really sorry of bothering you if the answer is in the manpages, but I 
was unable to spot it. Also, I hope it is fine I posted here despite not 
being a libfabric developer myself.
Paolo Inaudi