[libfabric-users] feature requests

[Oucharek, Doug S]

We just recently added Multi-Rail support to LNet (Lustre networking layer).  We designate a “primary address” which is mapped to a list of addresses for Multi-Rail.  The application only uses the primary address to send/receive from the peer.

We then round-robin messages over that pool (taking into account NUMA and core affinity).  If an endpoint goes down, we just remove it from the pool and add it back in when it recovers.  If/when you start messing with core and NUMA affinity, things get complicated fast.  If you want to do resends in your Multi-Rail solution (so messages don’t fall on the floor when an endpoint goes down), you will need to implement that infrastructure.  Plan these two items out well ahead of time!  

Also, you have to decide if you want to mix providers in the same Multi-Rail bond.  We did this but it adds some curve-balls you need to manage.

Testing of both the configuration API and operation is not fun and will dramatically expand the size of your testing matrix.

Doug

> On Jun 6, 2017, at 4:36 PM, Hefty, Sean <sean.hefty at intel.com> wrote:
> 
>> If MPI or other middle layer is to implement the multirail, why bother
>> even with OFI: implement directly on the device and no need for extra
>> OFI overhead.
> 
> Well, if MPI wants to write to device registers, I say go for it.  Maybe write to assembly too, to avoid the extra C overhead.  :)
> 
> More seriously, I'd like to start by analyzing what it would take to add multi-rail support over reliable-datagram endpoints, with the assumption that this would provide similar performance to what MPI could do.  This has the added benefit that it could work with completely different networks, though I'm not sure if that's a requirement.
> 
> I guess the first thing to figure out is how addressing works when multi-rail is in use.  Would we need some sort of super-address that's a union of the underlying fabric addresses?
> _______________________________________________
> Libfabric-users mailing list
> Libfabric-users at lists.openfabrics.org
> http://lists.openfabrics.org/mailman/listinfo/libfabric-users