[libfabric-users] libfabric network atomic operations, processor atomic ops, and coherence

[Greg Titus]

Hi all --

I could get a fabric-specific version of this question answered within Cray Inc.  But I thought there might be interest in the general version, so I’m asking here after doing a quick check of the last 2 years of the archives and not seeing it discussed.

A network that implements network atomic operations may do so such that those ops are not coherent with processor atomic ops (including even simple READs) to the same memory locations.  This is true of Cray Inc.’s Aries interconnect, for example.  On an Aries-based Cray XC system, for a given memory location one has to either do all atomics ops using the NIC, or do all of them using the processor, or use both the NIC and the processor but take additional (well-defined) actions to ensure coherence between the two.  What can a programmer using libfabric atomics assume about coherence between network and processor atomic operations when running on such a network?  Is it incumbent upon providers supporting such networks to create the appearance of coherence?  Or is it incumbent upon the programmer to know about possible non-coherence and deal with it, and if so, does libfabric give any help with this?  I’ve looked through the man pages and don’t see any discussion of atomic coherence, either in fi_atomic(3) or fi_gni(7).

I’m interested in this because I’m currently developing a libfabric-based implementation of the multi-node communication layer of the runtime library for Chapel (https://chapel-lang.org/).  Chapel has atomics, and our existing uGNI-native comm layer has to take into account the processor/network atomic non-coherence on Gemini- and Aries-based Cray XE and XC systems.  We’d like the new libfabric comm layer to work with as wide a variety of providers as possible, so we’d like to take the most general approach to the coherence problem we can.

Thanks!

Greg Titus
Chapel core team