[libfabric-users] multi-receive CQ entry with FI_MULTI_RECV, not the last one read by fi_cq_read()?
gregory.titus at hpe.com
Tue Jun 23 13:08:16 PDT 2020
I post a multi-receive buffer via fi_recvmsg(..., FI_MULTI_RECV) and then process it by reading CQ entries which refer to the messages landing there. When I see a CQ entry with FI_MULTI_RECV in its flags I re-post that same multi-receive buffer with fi_recvmsg() again. Based on reading the man pages, I've put an assertion in my CQ entry processing that if an entry has FI_MULTI_RECV in its flags, that entry must be the last one my fi_cq_read() read. Essentially, this assertion is to confirm my understanding that nothing can be placed in the multi-receive buffer after the provider releases it.
Today, for the first time ever, my assertion failed. What is it that I've misunderstood?
Some possibly helpful information: this is in provider-agnostic middleware (the Chapel ofi-based communication layer), though if it helps, the provider in use was 'verbs;ofi_rxm'. The multi-receive buffer management and CQ processing is done entirely in one thread, so there's no-multi-threading involved at least in "user" code.
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Libfabric-users