[libfabric-users] Connectionless RDM question

[Hefty, Sean]

> What is the recommended way for B to get the data from A. It has not
> added the address of A to its address vector and has no fi_addr_t for A
> and when it receives a message from A – so how does it know who A is?
> 
> 
> 
> Does A have to send (on first connection/message) a warning message to
> B so that B can add A it to its AV first? Is this how it is normally
> done? (this would mimic the connect request/connected model used for
> MSG endpoint).

This makes sense and is a reasonable solution.  Conceptually, this is what libfabric does internally when it implements RDM EPs over providers that do not support RDM natively.

> Or … should every node add every other node to its AV at startup so
> that when B gets a pull request from A it at least has an fi_addr_t of
> A and only needs to know which node to pull from. (If I have thousands
> of nodes communicating, but only using a nearest neighbour pattern, it
> seems a waste to add N^2 addresses to AV at startup)

This is similar to what MPI does, and MPI has the same issue that it may never communicate with all peers.  MPI uses out of band job startup to gather and exchange addresses with all peers.

> When I receive a message on an RDM endpoint, the same endpoint handles
> all peers, so now I don’t know how to get the address of the message
> origin again – do I have to send the address with every rdma request so
> that the receiver know where to read from?

I didn't follow this.  Are you assuming that the local AV does or may not have the peer's address?

> PS. Extra question. If Node A registers node C in its AV and node B
> registers node C in its AV, will A dna B have the same fi_addr_t for C?

Not necessarily.  You can force this by using FI_AV_INDEX, which will map fi_addr_t to a simple integer index value.  Then both peers only need to insert node C's address into the AV at the same point -- say the first address.

> (Can they be shared? – imlying that I could send the fi_addr_t as a
> handle with my rdma read requests so that B would know what address to
> use when getting data)

- Sean