[ofiwg] fi_msg : Is it possible to receive partial data from the message?

[Ioannis Charalampidis]

Hello Sean,
I see your point. However I am also writing a middleware and I would
like to avoid 'memcpy's as much as possible. Therefore I wanted to
avoid the FI_MULTI_RECV solution, even though it looks indeed elegant.

I was thinking to do a single malloc only when I know the size of the
incoming buffer, then pass it to libfabric to receive the message and
then move it to the user. But this matches with your 'inefficient'
approach, which I totally agree. But in my case, the messages can be
as big as 50Mb, which translates to big memcpy cost.

My next approach will be to ask the user to provide me with a
'maximum' size of the expected message, and allocate buffers of that
size during the reception... indeed it doesn't sound the best, but I
can avoid copying data around.

If I understand correctly, in order to receive any kind of data data
with libfabric I will need to:

1) Post the receive buffers (with fi_recv*)
2) Wait for a receive CQ event

However I was wondering... is there any way to wait for a 'data
pending' event, containing the size of them? Or perhaps another kind
of channel through I can send some lightweight (64-bit long) messages?

Cheers,
Ioannis

On 21/01/16 16:30, Hefty, Sean wrote:
>> Therefore I wanted to ask, if there is there any way to: 1) Wait
>> for CQ when the message arrives 2) Peek the first few bytes
>> (header) 3) Then read the entire buffer
> 
> Libfabric is message based, not stream based, and buffering is the
> responsibility of the app.  The behavior that you're asking for
> would result in receive side data copies and is not supported.  The
> receive side buffering must be provided before the send is
> initiated, or the incoming send will be dropped/truncated.
> 
> There are a couple ways to handle this.  A less efficient way is to
> post a bunch of buffers large enough to receive any incoming
> message.  A more efficient way is to use FI_MULTI_RECV to post a
> single large buffer that will be used to pack received messages
> into the larger buffer.
> 
> There shouldn't be a need to carry the message size in an app
> header, as the completion will report the size of the message that
> was received.  But if you really need to implement the above
> behavior, it would need to be done as an abstraction over some
> method like one mentioned.
> 
> - Sean
>