[libfabric-users] multi-receive CQ entry with FI_MULTI_RECV, not the last one read by fi_cq_read()?

[Titus, Greg]

Greetings --

Following up my own question from yesterday, I now have some more information and a hypothesis about what happened.

I managed to reproduce the assertion failure and collect information about the CQ entries returned from fi_cq_read(). There were 5 of them, indexed 0 through 4. Entries 0,2,3,4 were for received messages (flags==FI_MSG|FI_RECV), and entry 1 was for the multi-receive buffer being released (flags=FI_MULTI_RECV).  Most strikingly, while the buf values in entries 2,3,4 had increasing addresses in my multi-receive buffer, those addresses were all lower than the buf value in entry 0.

I didn't mention it in my original email, but I had asserted FI_ORDER_SAS on both the sending and receiving endpoints. My hypothesis is that what I saw is a normal and reasonable side effect of that. I expect that the message associated with CQ entry 0 was placed in the buffer last, causing it to reach its fill limit and be released. But the provider determined that SAS message ordering required the CQ entries for the received messages to be presented in the order they were, and it also wanted to put the buffer-release CQ entry right after the message that caused it to happen.

And thereby hangs our tale, as they say.

greg


________________________________
From: Libfabric-users <libfabric-users-bounces at lists.openfabrics.org> on behalf of Titus, Greg <gregory.titus at hpe.com>
Sent: Tuesday, June 23, 2020 2:08 PM
To: libfabric-users at lists.openfabrics.org <libfabric-users at lists.openfabrics.org>
Subject: [libfabric-users] multi-receive CQ entry with FI_MULTI_RECV, not the last one read by fi_cq_read()?

I post a multi-receive buffer via fi_recvmsg(..., FI_MULTI_RECV) and then process it by reading CQ entries which refer to the messages landing there. When I see a CQ entry with FI_MULTI_RECV in its flags I re-post that same multi-receive buffer with fi_recvmsg() again. Based on reading the man pages, I've put an assertion in my CQ entry processing that if an entry has FI_MULTI_RECV in its flags, that entry must be the last one my fi_cq_read() read. Essentially, this assertion is to confirm my understanding that nothing can be placed in the multi-receive buffer after the provider releases it.

Today, for the first time ever, my assertion failed. What is it that I've misunderstood?

Some possibly helpful information: this is in provider-agnostic middleware (the Chapel ofi-based communication layer), though if it helps, the provider in use was 'verbs;ofi_rxm'.  The multi-receive buffer management and CQ processing is done entirely in one thread, so there's no-multi-threading involved at least in "user" code.

Thanks!

greg

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/libfabric-users/attachments/20200624/5ab9f948/attachment.htm>