[libfabric-users] multi-receive CQ entry with FI_MULTI_RECV, not the last one read by fi_cq_read()?

[Hefty, Sean]

> I called fi_cq_read(cq, buf, 5) and it filled in buf with 5 entries.
> buf[0].flags==FI_MSG|FI_RECV and buf[0].buf was near the end of the multi-receive
> buffer.  buf[1].flags==FI_MULTI_RECV.  buf[2].flags==FI_MSG|FI_RECV and buf[2].buf was
> much earlier in the multi-receive buffer than buf[0].buf.  buf[3] and buf[4] were the
> same as buf[2] but with different .buf pointers though both were also much earlier in
> the multi-receive buffer than buf[0].buf.

Are you sure that the buffer was only posted once?  When is the buffer reposted?

The scenario you're describing is possible if the buffer is posted twice, or if a single buffer were split and posted as two receives.  For example:

If buf[0].buf + buf[0].len is near the end of the buffer...

buf[1] would mark the buffer as done (FI_MULTI_RECV set)

buf[2].buf would then reference the start of the buffer again
buf[3].buf would be near buf[2].buf + buf[2].len
buf[4].buf ~ buf[3].buf + buf[3].len