[ofiwg] Using libfabric on Titan

[Philip Davis]

Thanks for the quick reply. That’s disappointing, but understandable. Is the GNI provider supported on Cori and Theta?

> On May 3, 2018, at 2:49 PM, Pritchard Jr., Howard <howardp at lanl.gov> wrote:
> 
> the GNI provider isn’t supposed on Titan - OS is too old.
> 
> 
> -- 
> Howard Pritchard
> 
> B Schedule
> HPC-ENV
> Office 9, 2nd floor Research Park
> TA-03, Building 4200, Room 03U
> 
> Los Alamos National Laboratory
> 
> 
> 
> 
> 
> On 5/3/18, 11:21 AM, "Hefty, Sean" <sean.hefty at intel.com> wrote:
> 
>> Copying ofiwg, Howard, and Jim.
>> 
>> 
>>> Apologies if this has been asked before.
>>> 
>>> I am trying to use the GNI provider for libfabric on Titan, the XK7
>>> machine at ORNL. Is this possible/supported?
>>> 
>>> More specifically, I need to be able to communicate between aprun
>>> instances (e.g. aprun ./writer and aprun ./reader are able to
>>> communicate with each other) so the sharing of credentials between the
>>> programs is necessary. I have been referring to the cray-tests repo as
>>> an example of how to use rdma credentials for Cray in Libfabric. I
>>> modified this code somewhat by replacing all DRC code with just
>>> setting the cookie manually, and had hoped that would work.
>>> Unfortunately, the library relies on GNI_GetPtag to set the ptag, and
>>> this API call does not seem to work on Titan. I can get the
>>> auth_keys_test to pass by adding a ptag field to the
>>> fi_gni_raw_auth_key struct along with appropriate language to handle
>>> that, but I was wondering if there was some inherently supported way
>>> to do that same thing without modifying libfabric.
>>> 
>>> Other than credentials exchange/loading, i am wondering what I can
>>> expect in trying to use the GNI provider on a non-ARIES system. Has
>>> anyone done this successfully?
>>> 
>>> Thanks,
>>> Philip
>