[ofiwg] Hugepages usage in libfabric

[Jason Gunthorpe]

On Wed, Apr 10, 2019 at 02:36:53PM +0000, James Swaro wrote:
>    I’d be interested in exposing an environment variable, flag, or general
>    tunable in libfabric for applications to indicate whether they want to
>    use huge pages. Using the verbs provider on an internal development
>    system, I’ve run into an issue where use of huge pages makes the memory
>    registration function fail unless RDMAV_HUGEPAGES_SAFE is set.
>    See [1]https://lists.openfabrics.org/pipermail/ewg/2010-July/015609.htm
>    l for context.

Oh gross, this should just be fixed in verbs to not require the ugly
environment variable in the first place.

>    general, if necessary. For verbs, use of the RDMAV_HUGEPAGES_SAFE
>    variable tends to impact performance mostly after the rendezvous
>    threshold. After reaching the rendezvous threshold, latency for data
>    transfer operations appears to increase by a factor of ten.

If the app isn't forking it shouldn't even be calling ibv_fork_init in
the first place, which solves both problems.

Jason