[libfabric-users] Determining available providers

[Jeff Squyres (jsquyres)]

On Sep 26, 2018, at 10:04 AM, Philip Davis <philip.e.davis at rutgers.edu> wrote:
> 
> I appreciate the quick reply. A fair part of my interest is in setting sensible defaults at compile time, but perhaps that logic should be pushed out to runtime.

Speaking from my experience in the Open MPI project: run-time decisions are what we find are the most flexible.  It costs you more code to do that, but we have found that users, system administrators, and packagers really prefer having a software package that can be built once and used in many different ways.  I don't know what your goals are for this software, but if you can invest the time to build that additional infrastructure (and you don't really even have to get that fancy), it will result in a longer-lived project (IMNSHO).

In Open MPI, we use the plugin concept -- i.e., independent DSOs that are dlopen()ed at runtime.  Open MPI's configure script aggressively searches for as many header files/libraries that it can find for all kind of different packages.  If it finds a given header/library combo, it builds one or more plugins to support that library (e.g., libfabric, libibverbs, ...etc.).  Libfabric's configure script is similar, but not quite as aggressive (and doesn't setup to build DSOs by default).  Download both Open MPI and Libfabric and actually read the stdout of their configure scripts and you'll see what I mean.

In this way, we encourage packagers to have as many of the dependent libraries/headers available when building.  E.g., the Open MPI / libfabric binary packages in various Linux distros actually contain support for many different networking / run-time environments.

Then at run-time, Open MPI dlopen()s the plugins.  If a given dlopen() fails, it's not an error -- because perhaps this is a different machine than the build machine and some dependent library is not present.  No problem -- skip that plugin and move on.

If the plugin successfully dlopen()s, then we query it and ask it if it wants to run.  E.g., we query libfabric and say "hey, are there any libfabric devices on this machine?"  If the answer is "no", then we dlclose() the plugin and move on (again: it's not an error -- it's just that this plugin apparently isn't relevant to this system).

Like I mentioned above: this costs you software engineering and code for this kind of infrastructure.  But it gives you flexibility at runtime that is very useful.  Real-world example: run your code against network A, and then run the same build of your code again against network B.  Then you have an exact apples-to-apples comparison of networks A and B.

If it helps, here's a writeup of Open MPI's plugin architecture: http://www.aosabook.org/en/openmpi.html

-- 
Jeff Squyres
jsquyres at cisco.com