[libfabric-users] How to know which buffer corresponds to a completion entry

[Hubert Hirtz]

Hello!

I'm trying to develop an "echo" server that sends back messages it 
receives, and a netcat-like client that sends a message for each line on 
stdin.

I'm working with RDM, and the server has one "recv" buffer.  For every 
message it gets, it picks a "send" buffer from a bank of buffers, memcpy 
the "recv" buffer to it and call "fi_send" to push the "send" buffer to 
the provider.

The problem is, when the completion queue yields the server has no idea 
what "send" buffer is now free to use.

I'm using "FI_CQ_FORMAT_MSG" with "fi_cq_readfrom" to know which client 
the server should respond to, but the address is always 0 when the 
completion is flagged "FI_SEND".  I also tried to use 
"FI_CQ_FORMAT_DATA" to match the buffer address with the one from the 
completion entry, but "FI_SEND"-flagged entries have "buf == null".

Tested with verbs;ofi_rxm and UDP;ofi_rxd.

Should I create a completion queue for each client?  Or is there a way 
for the server to know which buffer can be reused after a completion?

Take care,
Hubert Hirtz