[libfabric-users] Question about FI_THREAD_SAFE and other threading models...

[Hefty, Sean]

> The definition of FI_THREAD_SAFE in the fi_domain(3) man page says:
> 
> FI_THREAD_SAFE
> 	A thread safe serialization model allows a multi-threaded application to access
> any allocated resources through any interface without restriction. All providers are
> required to support FI_THREAD_SAFE.
> 
> That appears to mean that all providers are required to handle any
> serialization/locking that might be required and that the application can be as multi-
> threaded as desired...?

Correct -- that is the intent.

> If that's the case, then why do we bother with more restrictive serialization models
> like FI_THREAD_ENDPOINT? Wouldn't they be redundant?

If the application is single threaded or assigns fabric resources to threads (no sharing), providers can eliminate locks.  The app indicates this through the threading attribute.  Some of the providers do this internally.  As an example, the base endpoint implementation does not provide locking unless the app requires thread safety:

	if (ep->domain->threading != FI_THREAD_SAFE) {
		ep->lock_acquire = ofi_fastlock_acquire_noop;
		ep->lock_release = ofi_fastlock_release_noop;
	} else {
		ep->lock_acquire = ofi_fastlock_acquire;
		ep->lock_release = ofi_fastlock_release;
	}

The base completion queue implementation is similar, but more restrictive as to when locks can be removed:

	if (cq->domain->threading == FI_THREAD_COMPLETION ||
	    (cq->domain->threading == FI_THREAD_DOMAIN)) {
		cq->cq_fastlock_acquire = ofi_fastlock_acquire_noop;
		cq->cq_fastlock_release = ofi_fastlock_release_noop;
	} else {
		cq->cq_fastlock_acquire = ofi_fastlock_acquire;
		cq->cq_fastlock_release = ofi_fastlock_release;
	}

- Sean