[ofiwg] Clarification of definition of FI_THREAD_DOMAIN

[Xiong, Jianxin]

Your understanding is correct.  The recommendation is based on how feasible to have a lockless implementation in the providers. That also matches with how middleware like MPI is doing today.

Does FI_THREAD_COMPLETION fits better with the multi-threaded RMA use case? It is recommended for scalable endpoints because that's when this threading model is more likely supported by the provider.  But using that with regular endpoint is totally fine if available.

For simplicity at the user end, maybe just go with FI_THREAD_SAFE.

-Jianxin

From: ofiwg <ofiwg-bounces at lists.openfabrics.org> On Behalf Of Byrne, John (Labs)
Sent: Wednesday, June 5, 2024 10:10 AM
To: ofiwg at lists.openfabrics.org
Subject: [ofiwg] Clarification of definition of FI_THREAD_DOMAIN

FI_THREAD_DOMAIN
A domain serialization model requires applications to serialize access to all objects belonging to a domain.

My immediate take on this definition is that if I am multi-threading I have to have a single lock that I use to access any object belonging to a fi_domain instance; which seems like a terrible idea for multi-threading. However, in Jianxin's 2.0 API update at the workshop https://www.openfabrics.org/wp-content/uploads/2024-workshop/2024-workshop-presentations/session-1.pdf, it says: "Recommend FI_THREAD_DOMAIN for multi-thread app with regular endpoint."  If my interpretation of the meaning of FI_THREAD_DOMAIN is correct, then the only way this makes sense to me is for the expectation to be that a unique fi_domain instance and endpoint be created for each thread. Is this correct or is there something I'm misunderstanding? If it is correct, then there are some painful implications for multi-threading RMA.

Thanks,

John Byrne


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/ofiwg/attachments/20240605/15ae795a/attachment.htm>