[libfabric-users] multi-receive CQ entry with FI_MULTI_RECV, not the last one read by fi_cq_read()?

[Titus, Greg]

Hmm, interesting.  My only ordering assertions are that I have FI_ORDER_RMA_RAW|FI_ORDER_SAS|FI_ORDER_SAW in both the tx_attr and rx_attr msg_order members of the hints for my fi_getinfo(). In the tx_attr and rx_attr op_flags I've just got FI_COMPLETION. But the CQ entry with the FI_MULTI_RECV flag set was entry 1 out of a range of 0..4, as I said. Definitely not the last one.

greg

________________________________
From: Hefty, Sean <sean.hefty at intel.com>
Sent: Wednesday, June 24, 2020 1:39 PM
To: Titus, Greg <gregory.titus at hpe.com>; libfabric-users at lists.openfabrics.org <libfabric-users at lists.openfabrics.org>
Subject: RE: multi-receive CQ entry with FI_MULTI_RECV, not the last one read by fi_cq_read()?

> I sent a follow-up this morning, which you may not have seen yet.  In that I
> hypothesized that this was due to the CQ entry ordering not being the same as the
> message ordering in the buffer because I'd thrown FI_ORDER_SAS on both the sending and
> receiving endpoints, and the FI_MULTI_RECEIVE CQ entry was effectively in buffer order
> rather than CQ order.

Completion order is separate from data order and not guaranteed unless FI_ORDER_STRICT has been enabled.  I don't know how you structured your assert, but the completion with FI_MULTI_RECV flag set should be last, as it indicates that the buffer is safe to repost.

> To answer your other questions, this was with the 1.10.1 release tarball and I'm only
> using one multi-receive buffer at a time.

The fixes I was referring to are in that release.

- Sean
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/libfabric-users/attachments/20200624/9c757a1c/attachment.htm>