[libfabric-users] multi-receive CQ entry with FI_MULTI_RECV, not the last one read by fi_cq_read()?

[Titus, Greg]

I post a multi-receive buffer via fi_recvmsg(..., FI_MULTI_RECV) and then process it by reading CQ entries which refer to the messages landing there. When I see a CQ entry with FI_MULTI_RECV in its flags I re-post that same multi-receive buffer with fi_recvmsg() again. Based on reading the man pages, I've put an assertion in my CQ entry processing that if an entry has FI_MULTI_RECV in its flags, that entry must be the last one my fi_cq_read() read. Essentially, this assertion is to confirm my understanding that nothing can be placed in the multi-receive buffer after the provider releases it.

Today, for the first time ever, my assertion failed. What is it that I've misunderstood?

Some possibly helpful information: this is in provider-agnostic middleware (the Chapel ofi-based communication layer), though if it helps, the provider in use was 'verbs;ofi_rxm'.  The multi-receive buffer management and CQ processing is done entirely in one thread, so there's no-multi-threading involved at least in "user" code.

Thanks!

greg

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/libfabric-users/attachments/20200623/dd74062f/attachment.htm>