[libfabric-users] Using libfabric on Titan

[Hefty, Sean]

Copying ofiwg, Howard, and Jim.


> Apologies if this has been asked before.
> 
> I am trying to use the GNI provider for libfabric on Titan, the XK7
> machine at ORNL. Is this possible/supported?
> 
> More specifically, I need to be able to communicate between aprun
> instances (e.g. aprun ./writer and aprun ./reader are able to
> communicate with each other) so the sharing of credentials between the
> programs is necessary. I have been referring to the cray-tests repo as
> an example of how to use rdma credentials for Cray in Libfabric. I
> modified this code somewhat by replacing all DRC code with just
> setting the cookie manually, and had hoped that would work.
> Unfortunately, the library relies on GNI_GetPtag to set the ptag, and
> this API call does not seem to work on Titan. I can get the
> auth_keys_test to pass by adding a ptag field to the
> fi_gni_raw_auth_key struct along with appropriate language to handle
> that, but I was wondering if there was some inherently supported way
> to do that same thing without modifying libfabric.
> 
> Other than credentials exchange/loading, i am wondering what I can
> expect in trying to use the GNI provider on a non-ARIES system. Has
> anyone done this successfully?
> 
> Thanks,
> Philip