[openib-general] mwrite64 - need for uar object in access layer

[Michael S. Tsirkin]

Hello!
Roland, just looking at mwrite64/mread64 I would like to point out
that you can avoid the whole complexity and the need to write 64 bit
atomically for a doorbell if you simply allocate a separate UAR
to the resource that you want to ring the doorbell on.

If you do this, tavor hardware will be able to detect and handle
the case where two doorbells are inter-mixed.

For example if you give a separate UAR to each EQ you'll be able
to avoid these spinlocks in interrupt path.
The cost of UAR is not too high - by default we have normally
2K UAR pages available - so I think, its OK to give
IP over IB its own UAR for its QP/CQ and then it wont need
any atomicity, either.


For things like SDP I think it makes sence to have UAR,
say, per process.

Looking at the non-SSE, 32 bit path, I would think you want
3 spinlocks per UAR (for CQ, Send Q and Receive Q),
and not a global doorbell_lock. Granted, I dont know
how relevant any 32-bit non-SSE architectures are ...

I conclude the access layer needs some interface to manage
UARs. It might seem tavor specific but I think something
like UAR is what everyone has to do anyway to have OS bypass, right?

Opinions?

MST