[libfabric-users] Tagged message question

[Biddiscombe, John A.]

Sean et al


I'm just resurrecting an old thread for clarification


>If the transfer is large, it won't complete immediately and the buffering will occur on the send side.  Only the tagged information will be transferred.  Once the receiver posts a receive with the correct tag, the data will be retrieved.


Does this mean that if I send a large tagged message (What is the threshold?, then only the tag info is sent initially and

a) if a posted receive buffer with a tag match is already present, the receiving side will do an RMA get from the send side buffer to the receive buffer (and then both sides get a completion)

b) if no receive buffer is posted yet, the send side will not post a completion, but when the receive side does post a matching buffer, an RMA get will finish the job and both sides complete.


The reason I ask is because I handle larger messages (>threshold) using RMA - but if libfabric is doing this internally, I can drop this completely for tagged messages and simply use fi_trecv and fi_tsend regardless of size.


what I really mean is - Is there any advantage to me sending the memory registration info in a small (unexpected) message and then doing an RMA get from the remote side myself when the tagged receive is setup?

Thanks

JB
________________________________
From: Hefty, Sean <sean.hefty at intel.com>
Sent: 16 March 2020 17:53
To: Chris Dolan; Biddiscombe, John A.
Cc: libfabric-users at lists.openfabrics.org
Subject: RE: [libfabric-users] Tagged message question

>        Who provided the memory that the received message went into between being
> received and my posting the recv buffer? Did libfabric supply some memory buffer 'on
> the fly' that the tagged message went into?

Adding to Chris' comments, these are known as 'unexpected messages'.  They are a common MPI problem and nearly impossible to avoid, even for well-designed apps.

Most commonly available hardware lack the ability to perform tag matching.  As a result, rxm buffers all receives before deciding how to process it.  If you're using RDM endpoints, there's really no additional cost for using tagged transfers, versus untagged.

If the transfer is large, it won't complete immediately and the buffering will occur on the send side.  Only the tagged information will be transferred.  Once the receiver posts a receive with the correct tag, the data will be retrieved.

- Sean
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/libfabric-users/attachments/20200730/2b566cb1/attachment.htm>