<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
This is middleware, so much is not known. We don't know until we choose one at execution time what the provider will be, and we don't have any knowledge whatsoever about AMO vs. RMA mix; that's determined by the behavior of the user program which is written
in Chapel and linked with the runtime. So we're definitely not designing the runtime, or even just the part of it that handles remote communication, around the NIC or the provider. Rather, the runtime has to be written to produce correct Chapel semantics
no matter which network and/or provider it finds itself using. Also note that this is only one implementation of the so-called "comm layer" part of the runtime; we also have implementations based on GASNet and, for the Cray Gemini- and Aries-based systems,
the native uGNI library. Which implementation you will use is selected when the runtime is built, which is typically long (perhaps years) before the user program is compiled and linked with it.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
There is already code both in the runtime and elsewhere in the Chapel SW stack to choose to do remote atomic ops using CPU AMOs via Active Messages instead of doing true network AMOs. We make this decision as early as possible, just to marginally improve performance.
If the configuration is one that definitely cannot support network AMOs (pre-GASNetEX based communication, for example) then we effectively make the decision when the user program is being compiled. If the configuration can support network AMOs but the specifics
of the situation prevent it (Cray Aries network with the uGNI-based comm layer, but the target object happens not to be in registered memory) then we make the decision during execution. We automatically get correct ordering for AM-mediated CPU AMOs, because
the Active Messages we use for these are fully blocking -- they don't return a 'done' indicator to the initiator until the AMO is complete.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
greg</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Hefty, Sean <sean.hefty@intel.com><br>
<b>Sent:</b> Tuesday, February 11, 2020 11:11 AM<br>
<b>To:</b> Titus, Greg <gregory.titus@hpe.com>; libfabric-users@lists.openfabrics.org <libfabric-users@lists.openfabrics.org><br>
<b>Subject:</b> RE: libfabric transaction ordering w.r.t. Chapel memory consistency model</font>
<div> </div>
</div>
<div class="BodyFragment"><font size="2"><span style="font-size:11pt;">
<div class="PlainText">> Are you wanting the initiator or the target to know that the data is visible?<br>
> For the target, verbs indicates that the data is visible when a completion is read at<br>
> the target for an operation that followed the write, or if the completion is for the<br>
> write itself (i.e. carries CQ data).<br>
> <br>
> <br>
> The initiator. The requirement here is that a task (Chapel's instance of sequential<br>
> execution) must observe the results of its own regular reads and writes to the same<br>
> address to have occurred in execution order. I.e., when a task writes to some location<br>
> and then reads from the same location, the value it reads must be the one that was<br>
> written. Similarly, when a task reads from and then writes to a location, the value<br>
> read must be what the location held before the write. Or more succinctly, within a<br>
> single task, regular reads and writes to the same location cannot be reordered. (This<br>
> is for data-race-free programs, so assume no other task is referencing this same<br>
> location during this period.)<br>
<br>
This is really an ordering requirement, not a visibility one. It does sound like you do need rxm to add support for fencing. That should be possible, eventually...<br>
<br>
> > ... I also need to ensure that<br>
> <br>
> > when a single task does an atomic op followed by a regular load or store, the<br>
> effect of<br>
> <br>
> > the atomic op on its target object is seen before the load or store references<br>
> memory.<br>
> <br>
> <br>
> ORDER_WAW orders both atomic updates and RMA write operations against each other.<br>
> ORDER_ATOMIC_WAW and ORDER_RMA_WAW allows specifying those separately. It sounds like<br>
> ORDER_WAW (etc.) is what you want.<br>
> <br>
> <br>
> Beyond saying that it doesn't support FI_FENCE (as discussed below), the fi_rxm man<br>
> page also says that if FI_ATOMIC is specified in the hints capabilities,<br>
> FI_ORDER_{RAR,RAW,WAR,WAW,SAR,SAW} support is disabled. It also doesn't include<br>
> FI_ATOMIC in the capabilities unless you specifically request it, which may well be<br>
> because of this limitation. So I'm pretty sure I'm going to be using processor atomics<br>
> done via Active Messages for remote atomic ops with ofi_rxm;verbs.<br>
<br>
Unfortunately, verbs devices have extremely limited support for atomic operations, so nearly all of them must be implemented by the CPU. And trying to keep ordering semantics between RMA and atomics would mean using the CPU, rather than the NIC, for RMA.<br>
<br>
> Thanks for all the feedback, Sean! Not all the answers make me happy from a<br>
> performance point of view, but at least it doesn't sound like I missed any better ways<br>
> of doing things than the ones I'd come up with.<br>
<br>
What I would look at is, is there a better performing option over verbs devices, given the semantics that you need? I don't think that there is generically. Maybe some optimizations are possible if you knew the traffic mix between atomics and RMA.<br>
<br>
You need to fence writes after reads to get WAR ordering. Verbs HW can do this. But having to fall back to CPU atomics would require a software based fence. IMO, it would be best to push that down into libfabric, so Chapel isn't designing around a specific
NIC implementation. I don't have a good answer for maintaining ordering between HW based RMA and CPU atomics. If the traffic mix is heavy on atomics, or the RMA transfers are usually small, software based RMA may end up being preferred.<br>
<br>
- Sean<br>
</div>
</span></font></div>
</body>
</html>