[ofiwg] [libfabric-users] same process, multiple endpoints

[Hefty, Sean]

Re-posting to ofiwg mail list, as this is more of a developer question.

> Hi all,
> 
> I am from the alpha group [http://alpha.di.unito.it/] (CS dept. @ Univ. of
> Torino, Italy).
> We just started to integrate libfabric into FastFlow [link], on the track
> of the great job by Paolo Inaudi with the A3Cube provider.
> 
> After several readings of all the man pages and some online tutorials, I
> still cannot understand how to work with contexts, domains and endpoints.

I'm not sure what you're referring to exactly by context.  In the documentation, a context is probably easiest to think of a command queue or work queue.  In the simplest case, an endpoint has 2 contexts -- one for sending and another for receiving.

Note that we use the term context rather than queue because there's no guarantee that work that has been posted to an endpoint completes in order.

> I try to illustrate the problem:
> 1) single process attached to NIC with address X
> 2) two connection-less endpoints A and B
> 3) endpoint A should accept data from X:portA, while B from X:portB
> 
> If I try to map the hierarchy proposed in the libfabric "object model" I
> would instantiate one context, one domain and both endpoints (and
> respective queues) to that single domain. In this setting, I cannot find
> how to specify the different ports for the endpoints. Actually I just need
> to ensure endpoint A will find only messages sent to portA in its
> completion queue, but I cannot figure out how to do it.

It sounds like you need different addresses for the two ports.  Then assign address A from port A to endpoint A, and address B from port B to endpoint B.

An example from IP land would be to assign port A address 192.168.1.1 and port B 192.168.2.1.  Endpoint A would have address 192.168.1.1:12345, and B address 192.168.2.1:12345.  The '12345' in this example are transport (e.g. UDP or TCP port) numbers.

> Conversely, all the examples I saw (e.g. fabtests benchmarks) use a single
> endpoint per process and the libfabric context is initialized with hints
> and arguments specific to the endpoint. This means I should prefer to
> instantiate one domain (and one context) for each endpoint?

If you have a single NIC, a single domain should be sufficient.

- Sean