[ofiwg] selecting a fabric for a destination

[Reese Faucette (rfaucett)]

"best" is too vague below.  To limit the scope of this, let's specifically call it "shortest distance" to make it clear we're not looking at speed, congestion, etc.  Simply an abstracted "network hops".

To put it in real terms - you have two interfaces, the remote is on the same subnet as one, but could also be reached through a router from the other.  You want to pick the local one, and you don't want the application having to parse kernel routing tables to find out which one to use, or even know that kernel routing tables are involved in the answer.  How does it pick the right one?

-reese


From: ofiwg-bounces at lists.openfabrics.org [mailto:ofiwg-bounces at lists.openfabrics.org] On Behalf Of Reese Faucette (rfaucett)
Sent: Wednesday, October 08, 2014 4:20 PM
To: ofiwg at lists.openfabrics.org
Subject: [ofiwg] selecting a fabric for a destination

If a program needs to send to only one remote address, selecting a fabric is simple: the program just passes the remote address to fi_getinfo and is returned an ordered list of fabrics that can communicate with the remote address.

Suppose, though, that you have an application that may need to send data to a large set of remote destinations that may only become known "later".  I see the app calling fi_getinfo() at startup to learn what suitable fabrics are available, and opening a domain and set of endpoints on each of them.

Now, later, through some out-of-band mechanism, the process learns of remote destinations it should contact.  The problem the app now faces is, "which endpoint do I use?" since not all fabrics can necessarily reach the remote address.  Calling fi_getinfo() with this newly learned address is somewhat clumsy, since fi_getinfo will malloc a whole new list of fi_infos to return, which then must be mapped by the application to actual fabric/domain/endpoint in use.

Any suggestions for how the app should learn which fabric is the best to use to reach this remote destination?  One possibility is that the program can pass in a list of fabrics and a remote destination to "fi_fabric_for_dest(fi_list, remote_address)" and be returned a pointer to the "best" one to use, "best" defined by the providers.

Another possibility is for the program itself to loop through the fabrics it knows about calling "fi_get_distance_metric(domain, remote_address)" and pick the one with the lowest (provider-defined within guidelines) metric.

The mechanism sockets uses is, IMO, unacceptable here - creating one "fabric" for all interfaces and letting the kernel (or some agent) pick the right one at send time.  Bypassing all this critical path lookup is much of the reason we want OS-bypass in the first place.

Thoughts, suggestions, flames?
-reese

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/ofiwg/attachments/20141008/f5b136f2/attachment.html>