[libfabric-users] Endpoints revisited

[Hefty, Sean]

> I managed to break my code whilst trying to improve it - in a nutshell, I have one
> endpoint that is configured to receive and one that is for transmit. At startup all
> ranks share the Receive addresses so that they can fill the address vector, but when I
> use FI_SOURCE - a send from rank 0 appears to come from a different address than the
> receive address of rank 0 (because 2 endpoints not 1).
> 
> I had this problem before when I implemented threadlocal endpoints and somehow I got it
> working, but the latest version of the code appears munged.
> 
> Is there any way of having the transmit endpoint use the same address as the receive
> endpoint on each node - so that  when a message is received on Node B from Node A, it
> will appear to come from the same address as Node B uses when sending to Node A?

Different endpoints will have different transport level addresses.  To have the same address, they transmit and receive contexts need to belong to the same endpoint.  Is there a reason why you separated them?

I'm guessing you could carry a 16-bit source rank in the message, or maybe place that into the CQ data.  This adds a small amount of extra data to carry, but should be fairly easy as long as all ranks have mirroring AVs.

- Sean