[openib-general] uverbs: pthread_mutex -> pthread_spinlock ?

[Roland Dreier]

    Michael> I am able to shave about 200ns off the rdma post latency,
    Michael> by using pthread_spinlock instead of pthread_mutex for
    Michael> protecting the qp post op in libmthca.

    Michael> I'm aware of course that a context switch when spinlock
    Michael> is held may waste a whole timeslice , but maybe for short
    Michael> operations such as this it's reasonable to use spinlocks?

You're right, this is a significant performance boost.  I had believed
that since pthread_mutex_lock and pthread_mutex_unlock can be done
completely in userspace with NPTL and futexes (with only a single
locked instruction when there is no contention), then doing
pthread_spin_lock/pthread_spin_unlock instead would be roughly
equivalent.  However, a quick synthetic benchmark shows that I was
completely wrong: uncontended pthread_mutex_t operations are much
slower than uncontended pthread_spin_t operations on i386, x86_64,
ppc64 and ia64.

I've committed a changeset for libmthca that converts from
pthread_mutex_t to pthread_spinlock_t, which results in a measurable
improvement on real pingpong tests.

Thanks,
  Roland