[ofiwg] Hugepages usage in libfabric

[James Swaro]

I'm not certain what prompted performing the check in ibv_fork_init. However, it seems to me that hugepages registration support is orthogonal to forking. Given what the environment variable addresses and the fact that the variable is only read in ibv_fork_init, I would wonder if the application would crash if it did NOT call ibv_fork_init -- even if it wasn't planning to fork. 

--- 
James Swaro 
Cray Inc. 
 

On 4/10/19, 10:03 AM, "Jason Gunthorpe" <jgg at ziepe.ca> wrote:

    On Wed, Apr 10, 2019 at 02:36:53PM +0000, James Swaro wrote:
    >    I’d be interested in exposing an environment variable, flag, or general
    >    tunable in libfabric for applications to indicate whether they want to
    >    use huge pages. Using the verbs provider on an internal development
    >    system, I’ve run into an issue where use of huge pages makes the memory
    >    registration function fail unless RDMAV_HUGEPAGES_SAFE is set.
    >    See [1]https://lists.openfabrics.org/pipermail/ewg/2010-July/015609.htm
    >    l for context.
    
    Oh gross, this should just be fixed in verbs to not require the ugly
    environment variable in the first place.
    
    >    general, if necessary. For verbs, use of the RDMAV_HUGEPAGES_SAFE
    >    variable tends to impact performance mostly after the rendezvous
    >    threshold. After reaching the rendezvous threshold, latency for data
    >    transfer operations appears to increase by a factor of ten.
    
    If the app isn't forking it shouldn't even be calling ibv_fork_init in
    the first place, which solves both problems.
    
    Jason