[ofa-general] Re: [patch 2/6] mmu_notifier: Callbacks to invalidate address ranges

[Christoph Lameter]

On Fri, 29 Feb 2008, Andrea Arcangeli wrote:

> > AFAICT The rw semaphore fastpath is similar in performance to a rw 
> > spinlock. 
> 
> read side is taken in the slow path.

Slowpath meaning VM slowpath or lock slow path? Its seems that the rwsem 
read side path is pretty efficient:

static inline void __down_read(struct rw_semaphore *sem)
{
        __asm__ __volatile__(
                "# beginning down_read\n\t"
LOCK_PREFIX     "  incl      (%%eax)\n\t" /* adds 0x00000001, returns the old value */
                "  jns        1f\n"
                "  call call_rwsem_down_read_failed\n"
                "1:\n\t"
                "# ending down_read\n\t"
                : "+m" (sem->count)
                : "a" (sem)
                : "memory", "cc");
}



> 
> write side is taken in the fast path.
> 
> pagefault is fast path, VM during swapping is slow path.

Not sure what you are saying here. A pagefault should be considered as a 
fast path and swapping is not performance critical?

> > > Perhaps the rwlock spinlock can be changed to a rw semaphore without
> > > measurable overscheduling in the fast path. However theoretically
> > 
> > Overscheduling? You mean overhead?
> 
> The only possible overhead that a rw semaphore could ever generate vs
> a rw lock is overscheduling.

Ok too many calls to schedule() because the slow path (of the semaphore) 
is taken?

> > On the other hand a semaphore puts the process to sleep and may actually 
> > improve performance because there is less time spend in a busy loop. 
> > Other processes may do something useful and we stay off the contended 
> > cacheline reducing traffic on the interconnect.
> 
> Yes, that's the positive side, the negative side is that you'll put
> the task in uninterruptible sleep and call schedule() and require a
> wakeup, because a list_add taking <1usec is running in the
> other cpu. No other downside. But that's the only reason it's a
> spinlock right now, infact there can't be any other reason.

But that is only happening for the contended case. Certainly a spinlock is 
better for 2p system but the more processors content for the lock (and 
the longer the hold off is, typical for the processors with 4p or 8p or 
more) the better a semaphore will work.