<div dir="ltr">HI Daniel,<div><br></div><div>Would you be able to write a small reproducer program for the problem you are observing</div><div>and open an issue on ofwig/libfabric? Also, does your app work with the sockets provider</div><div>and FI_PROGRESS_MANUAL? The sockets provider should work on your Cray system.</div><div><br></div><div>I think you are being a bit overly strict about the meaning of progress manual. Its not that</div><div>no data transfers can proceed until the app calls fi_cq_read and friends, but rather</div><div>that the app must make such calls to guarantee forward progress of previously posted</div><div>transactions it has made, as well as progression of data transfer operations from other</div><div>processes that initiated data transfer operations to EPs it has set up. </div><div><br></div><div>In the case of systems like Cray XC aries, where the underlying network can progress</div><div>RMA operations without software intervention (once they are posted to hw queues), it makes sense</div><div>to kick the GNI provider internal progress engine to keep submitting RMA requests to the</div><div>queues in the event there's a backlog of requests. The hw queues can only take</div><div>a certain number of requests - hence the reason for a possible backlog.</div><div><br></div><div>Howard</div><div><br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">2017-05-25 4:15 GMT-06:00 Daniel Deppisch <span dir="ltr"><<a href="mailto:danieldeppisch@onlinehome.de" target="_blank">danieldeppisch@onlinehome.de</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><u></u><div><div>Hello Howard,</div><div><br></div><div>thank you for your reply!</div><div><br></div><div>I have a small benchmarking app which gets stuck with control_progress=manual + data_progress=manual.</div><div>It does however work perfectly with control_progress=auto + data_progress=manual.</div><div>So I figured there would be control_progress missing.</div><div><br></div><div>If memory registration is synchronous, my guess would be that maybe AV-inserts are not progressed? Are those calls synchronous too? If they aren't, how do I get them to progress?</div><div>Or is there any other progress that might be missing?</div><div><br></div><div>The app gets stuck at the first attempt to wait for a transfer operation on a CQ. AV-insert calls do return.</div><div><br></div><div>On your question:</div><div>I don't really understand what is going on yet, so I can't really recommend anything. All I can say is that for me things are not working the way the manual suggests.</div><div>The manual tells me control_progress=manual is supported, but all ways to actually make that progress (at least the ones I can deduct from the manual) are not supported/implemented as the would involve the EQ.</div><div>I probably wouldn't implement unneccessary "EQ action" but rather update the manual for the GNI provider to explain this.</div><div><br></div><div>On your PR:</div><div>"<span style="line-height:21px;background-color:rgb(255,255,255)"><font size="3">it causes all data progress to be delayed until </font></span><span style="font-size:medium;line-height:21px;background-color:rgb(255,255,255)">the app starts calling fi_cq_read, etc."</span></div><div><span style="font-size:medium;line-height:21px;background-color:rgb(255,255,255)">From what I read in the manual, this is exactly what I would expect from data_progress=manual.</span></div><div><span style="font-size:medium;line-height:21px;background-color:rgb(255,255,255)"><br></span></div><div><span style="font-size:medium;line-height:21px;background-color:rgb(255,255,255)">Another note:</span></div><div><font size="3"><span style="line-height:21px;background-color:rgb(255,255,255)">Since I mentioned updating the manual, I just want to add that in my opinion it would be nice to have the manual explain that for the GNI provider, setting wait_objects to FI_WAIT_NONE is perfectly fine, even if you want to use synchronous calls on queues and counters. This is rather unexpected behaviour, as the general documentation (on CQ for example) states that you actually can't use blocking calls with FI_WAIT_NONE. I only found this info somewhat by accident somewhere in the depths of the GNI provider fork.</span></font></div><div><br></div><div><br></div><div><font size="3"><span style="line-height:21px;background-color:rgb(255,255,255)">Thanks and regards,</span></font></div><div><font size="3"><span style="line-height:21px;background-color:rgb(255,255,255)">Daniel</span></font></div><div><span style="font-size:medium;line-height:21px;background-color:rgb(255,255,255)"><br></span></div><div><span style="font-size:medium;line-height:21px;background-color:rgb(255,255,255)"><br></span></div>
<div>------ Originalnachricht ------</div>
<div>Von: "Howard Pritchard" <<a href="mailto:hppritcha@gmail.com" target="_blank">hppritcha@gmail.com</a>></div>
<div>An: "Daniel Deppisch" <<a href="mailto:danieldeppisch@onlinehome.de" target="_blank">danieldeppisch@onlinehome.de</a>></div>
<div>Cc: <a href="mailto:libfabric-users@lists.openfabrics.org" target="_blank">libfabric-users@lists.<wbr>openfabrics.org</a></div>
<div>Gesendet: 24.05.2017 17:16:47</div>
<div>Betreff: Re: [libfabric-users] GNI Provider manual control_progress</div><div><div class="h5"><div><br></div>
<div id="m_-931649895919960292xc426e6266bc34d8"><blockquote cite="http://CAF1Cqj5CSp9eixnK+Tjx-5n-JkxtVuCGD4a0c2yUFTVFeZwJUw@mail.gmail.com" type="cite" class="m_-931649895919960292cite2">
<div dir="ltr">Hello Daniel,<div><br></div><div>For the GNI provider, memory registration is currently a synchronous call. There is no need to poll</div><div>on an EQ to progress memory registration. From your reading of the man pages, do you think</div><div>that it would be less confusing if the GNI provider added support for using EQ events to progress</div><div>memory registration even when its not necessary? It would not be a big problem to add such </div><div>support.</div><div><br></div><div>For data transfer operations however, you will indeed need to poll on the send/recv counters/CQs</div><div>that have been bound to the EP initiating these data transfer operations.</div><div><br></div><div>Just a heads up that we have at least one PR that needs to get upstream that may impact your testing:</div><div><br></div><div><a href="https://github.com/ofi-cray/libfabric-cray/pull/1347" target="_blank">https://github.com/ofi-cray/<wbr>libfabric-cray/pull/1347</a><br></div><div><br></div><div>We should have that pushed upstream this week.</div><div><br></div><div>Howard</div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">2017-05-18 12:19 GMT-06:00 Daniel Deppisch <span dir="ltr"><<a href="mailto:danieldeppisch@onlinehome.de" target="_blank">danieldeppisch@onlinehome.de</a>></span><wbr>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><u></u>
<div>Hello,<div><br></div><div>im a master student from germany currently evaluating libfabric performance on a Cray Aries system.</div><div><br></div><div>I am struggling to get control_progress=FI_PROGRESS_M<wbr>ANUAL working.</div><div>Documentation clearly states it is supported for the GNI provider.</div><div><br></div><div>However, manual progress is described to be made when reading/waiting for the queue/counter where the completion will be reported.<br>For control operations this should be the event queue.</div><div><br></div><div>Now the GNI provider does not implement any of the functions for binding an EQ.</div><div>Neither can I bind control operation completions (like av_insert()s) to a CQ/counter using the cq_bind() flags.</div><div><br></div><div>If manual control progress is supported, but the eq cannot be used, how do I make manual control progress?</div><div><br></div><div>Thanks and regards,</div><div>Daniel Deppisch</div></div><br>______________________________<wbr>_________________<br>
Libfabric-users mailing list<br>
<a href="mailto:Libfabric-users@lists.openfabrics.org" target="_blank">Libfabric-users@lists.openfabr<wbr>ics.org</a><br>
<a href="http://lists.openfabrics.org/mailman/listinfo/libfabric-users" rel="noreferrer" target="_blank">http://lists.openfabrics.org/m<wbr>ailman/listinfo/libfabric-user<wbr>s</a><br>
<br></blockquote></div><br></div>
</blockquote></div>
</div></div></div></blockquote></div><br></div>