<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<blockquote style="border-left: 3px solid rgb(200, 200, 200); border-top-color: rgb(200, 200, 200); border-right-color: rgb(200, 200, 200); border-bottom-color: rgb(200, 200, 200); padding-left: 1ex; margin-left: 0.8ex; color: rgb(102, 102, 102);">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span>> ... how can I know when the result of a write is visible in<br>
</span>
<div>> the remote memory, so that I can retire the matching outstanding-write list entry?<br>
</div>
<div><br>
</div>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span>Are you wanting the initiator or the target to know that the data is visible? For the target, verbs indicates that the data is visible when a completion is read at the target for an operation that followed the write, or if the completion is for the write
itself (i.e. carries CQ data).</span><br>
</div>
</blockquote>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span>The initiator. The requirement here is that a task (Chapel's instance of sequential execution) must observe the results of its own regular reads and writes to the same address to have occurred in execution order. I.e., when a task
<span style="font-family: Calibri, Arial, Helvetica, sans-serif; background-color: rgb(255, 255, 255); display: inline !important">
writes to some location and then reads from the same location, the value it reads must be the one that was written</span>. Similarly, when a task <span style="font-family: Calibri, Arial, Helvetica, sans-serif; background-color: rgb(255, 255, 255); display: inline !important">reads
from and then writes to a location, the value read must be what the location held before the write</span>. Or more succinctly, within a single task, regular reads and writes to the same location cannot be reordered. (This is for data-race-free programs,
so assume no other task is referencing this same location during this period.)</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span><br>
</span></div>
<blockquote style="border-left: 3px solid rgb(200, 200, 200); border-top-color: rgb(200, 200, 200); border-right-color: rgb(200, 200, 200); border-bottom-color: rgb(200, 200, 200); padding-left: 1ex; margin-left: 0.8ex; color: rgb(102, 102, 102);">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span>> ... I also need to ensure that<br>
</span>
<div>> when a single task does an atomic op followed by a regular load or store, the effect of<br>
</div>
<div>> the atomic op on its target object is seen before the load or store references memory.<br>
</div>
<div><br>
</div>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span>ORDER_WAW orders both atomic updates and RMA write operations against each other. ORDER_ATOMIC_WAW and ORDER_RMA_WAW allows specifying those separately. It sounds like ORDER_WAW (etc.) is what you want.</span><br>
</div>
</blockquote>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span><br>
</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span>Beyond saying that it doesn't support FI_FENCE (as discussed below), the fi_rxm man page also says that if FI_ATOMIC is specified in the hints capabilities, FI_ORDER_{RAR,RAW,WAR,WAW,SAR,SAW} support is disabled. It also doesn't include FI_ATOMIC in
the capabilities unless you specifically request it, which may well be because of this limitation. So I'm pretty sure I'm going to be using processor atomics done via
<span style="font-family: Calibri, Arial, Helvetica, sans-serif; background-color: rgb(255, 255, 255); display: inline !important">
Active Messages </span>for remote atomic ops with ofi_rxm;verbs.</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span><br>
</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span>Thanks for all the feedback, Sean! Not all the answers make me happy from a performance point of view, but at least it doesn't sound like I missed any better ways of doing things than the ones I'd come up with.</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span><br>
</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span>greg</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span><br>
</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span><br>
</span></div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Hefty, Sean <sean.hefty@intel.com><br>
<b>Sent:</b> Monday, February 10, 2020 4:38 PM<br>
<b>To:</b> Titus, Greg <gregory.titus@hpe.com>; libfabric-users@lists.openfabrics.org <libfabric-users@lists.openfabrics.org><br>
<b>Subject:</b> RE: libfabric transaction ordering w.r.t. Chapel memory consistency model</font>
<div> </div>
</div>
<div class="BodyFragment"><font size="2"><span style="font-size:11pt;">
<div class="PlainText">> I'm implementing a libfabric-based multi-node communication module for the Chapel<br>
> (<a href=""></a>https://chapel-lang.org/
) runtime library. I have some questions about how best to<br>
> use libfabric's capabilities to implement Chapel's memory consistency model (MCM),<br>
> while nevertheless maximizing performance.<br>
> <br>
> Chapel's MCM is based on sequential consistency for data-race-free programs as adopted<br>
> by a number of other languages. In its simplest form, the MCM says that atomic ops<br>
> done by a task are seen to occur in program order with respect to both other atomic ops<br>
> and sequences of regular loads and stores done by that task, while regular loads and<br>
> stores are not ordered with respect to each other except that a task sees its own loads<br>
> and stores to the same address to have occurred in program order. It's basically the<br>
> UPC "non-strict" MCM, if you're familiar with that.<br>
<br>
I'm not familiar with non-strict MCM, and can't say that I fully understand the desired semantics being requested.<br>
<br>
>From the description, it sounds like you are wanting ORDER_RAW / WAR / and WAW, as you mention below.<br>
<br>
> For the following, assume I'm using FI_TRANSMIT_COMPLETE for completion, either by<br>
> default or explicitly. This might be either because FI_DELIVERY_COMPLETE isn't<br>
> available in some providers, or because transmit_complete performs better than<br>
> delivery_complete.<br>
<br>
Delivery complete requires a software based ack for most (all?) hardware.<br>
<br>
> For regular loads and stores, including the same-address clause, it seems I could make<br>
> libfabric match the Chapel MCM just by asserting<br>
> FI_ORDER_RAW|FI_ORDER_WAR|FI_ORDER_WAW. But that's overkill, because it will order<br>
<br>
It might seem like overkill, but probably isn't that bad if you consider what it takes to support these. You likely lose dynamic routing as a network feature. The verbs and tcp providers will give you WAW and RAW anyway.<br>
<br>
A more significant issue is that WAR ordering isn't supported natively by verbs hardware. You may need to fence write operations that follow reads to the same memory location.<br>
<br>
> transactions to all addresses, not just those that target the same address. So I could<br>
> also do something like maintaining a list of outstanding remote writes in each task and<br>
> consulting that for address matches to see if a later read refers to an earlier write<br>
> that is still in flight. That's no problem - it's a common technique for improving<br>
> performance. But if I do that, how can I know when the result of a write is visible in<br>
> the remote memory, so that I can retire the matching outstanding-write list entry? I<br>
<br>
Are you wanting the initiator or the target to know that the data is visible? For the target, verbs indicates that the data is visible when a completion is read at the target for an operation that followed the write, or if the completion is for the write itself
(i.e. carries CQ data).<br>
<br>
I don't recall if libfabric defines this level of visibility, but I don't think it does. That is likely a gap in the receive side completion semantics.<br>
<br>
If you need the initiator to know this, the API option is to issue an operation with delivery_complete with a fence flag (also like you mention). This is the best option we have to.day, but I'm looking at other possibilities here (in the context of persistent
memory).<br>
<br>
> believe I can force writes to complete remotely by setting up endpoints with<br>
> FI_ORDER_RAW and then doing dummy reads from each target I'm interested in. But that<br>
> seems heavyweight because it forces the writes to complete and really, I only want to<br>
> be informed when they complete, not force them to do so. It looks like FI_FENCE could<br>
> be used to solve this, but I'm not sure that's available to me because I think I need<br>
> to work with the verbs;ofi_rxm provider and ofi_rxm doesn't support FI_FENCE.<br>
<br>
Hmm... this sounds like a gap. I don't know why rxm doesn't just pass the fence flag through.<br>
<br>
> I have a similar issue with respect to ordering atomics. I need to ensure that the<br>
> effects on target objects of a sequence of atomic ops done by a single task are seen to<br>
> occur in program order. Would asserting FI_ORDER_ATOMIC_WAW on both initiating and<br>
> target endpoints guarantee order for target object updates? I also need to ensure that<br>
> when a single task does an atomic op followed by a regular load or store, the effect of<br>
> the atomic op on its target object is seen before the load or store references memory.<br>
<br>
ORDER_WAW orders both atomic updates and RMA write operations against each other. ORDER_ATOMIC_WAW and ORDER_RMA_WAW allows specifying those separately. It sounds like ORDER_WAW (etc.) is what you want.<br>
<br>
> The fi_atomic(3) man page only says that a completion isn't delivered at the originator<br>
> until after the result of a fetching atomic op is available there, and a completion (if<br>
> any) isn't delivered at the target until after the effect of an atomic op on its target<br>
> object is visible there. There doesn't seem to be a direct way to connect the change<br>
> to the target object and the delivery of a completion event to the initiator. So<br>
> what's the best way for an originator to ensure that the target effect of an atomic op<br>
> is visible before it continues? Do I need to do something like requesting remote<br>
> completions for atomic ops and have targets send messages back to initiators when they<br>
> see such events? That seems heavyweight.<br>
<br>
If you want the initiator to know that the data is visible at the target, then delivery_complete is the semantic that you want. However... rxm currently does not properly implement delivery_complete semantics. But if/when it does, it will require the use
of software acks above verbs devices. Yes, it's heavyweight, but it's the only option I'm aware of for those devices.<br>
<br>
> I also have to ensure that the effects of a sequence of regular loads and stores are<br>
> visible before the effect on the target of a subsequent atomic op, but I believe for<br>
> that I can use an extension of the solution for the same-address clause, and do a read<br>
> from every target I've written to since the last atomic op, with FI_ORDER_RAW asserted.<br>
> <br>
> (Note that some of the above may be moot for ofi_rxm, since at least in v1.8 it is<br>
> documented to disable support for some of the orderings if you ask for the FI_ATOMIC<br>
> capability. So for that one I may simply forego libfabric network atomics and use<br>
> processor atomics via Active Messages, a capability that already exists in Chapel<br>
> because of the need to be portable to less-capable networks.)<br>
<br>
I think this is a result of implementing the atomics in software, but still using RMA hardware when available.<br>
<br>
- Sean<br>
</div>
</span></font></div>
</body>
</html>