[openib-general] mwrite64 - need for uar object in access layer

[Roland Dreier]

    Michael> Hello!  Roland, just looking at mwrite64/mread64 I would
    Michael> like to point out that you can avoid the whole complexity
    Michael> and the need to write 64 bit atomically for a doorbell if
    Michael> you simply allocate a separate UAR to the resource that
    Michael> you want to ring the doorbell on.

    Michael> If you do this, tavor hardware will be able to detect and
    Michael> handle the case where two doorbells are inter-mixed.

    Michael> For example if you give a separate UAR to each EQ you'll
    Michael> be able to avoid these spinlocks in interrupt path.  The
    Michael> cost of UAR is not too high - by default we have normally
    Michael> 2K UAR pages available - so I think, its OK to give IP
    Michael> over IB its own UAR for its QP/CQ and then it wont need
    Michael> any atomicity, either.

    Michael> For things like SDP I think it makes sence to have UAR,
    Michael> say, per process.

    Michael> Looking at the non-SSE, 32 bit path, I would think you
    Michael> want 3 spinlocks per UAR (for CQ, Send Q and Receive Q),
    Michael> and not a global doorbell_lock. Granted, I dont know how
    Michael> relevant any 32-bit non-SSE architectures are ...

    Michael> I conclude the access layer needs some interface to
    Michael> manage UARs. It might seem tavor specific but I think
    Michael> something like UAR is what everyone has to do anyway to
    Michael> have OS bypass, right?

    Michael> Opinions?

Sounds interesting... I would be curious to see benchmarks for various
schemes.  Exposing UARs to consumers seems a little bit of a layering
violation but if it turns out to be a big win then it might be worth
it.  Of course I'm not sure how interested in optimizing 32-bit
architectures anyone is...

 - R.