[openib-general] mwrite64 - need for uar object in access layer
Roland Dreier
roland at topspin.com
Mon Sep 20 19:52:05 PDT 2004
Michael> Hello! Roland, just looking at mwrite64/mread64 I would
Michael> like to point out that you can avoid the whole complexity
Michael> and the need to write 64 bit atomically for a doorbell if
Michael> you simply allocate a separate UAR to the resource that
Michael> you want to ring the doorbell on.
Michael> If you do this, tavor hardware will be able to detect and
Michael> handle the case where two doorbells are inter-mixed.
Michael> For example if you give a separate UAR to each EQ you'll
Michael> be able to avoid these spinlocks in interrupt path. The
Michael> cost of UAR is not too high - by default we have normally
Michael> 2K UAR pages available - so I think, its OK to give IP
Michael> over IB its own UAR for its QP/CQ and then it wont need
Michael> any atomicity, either.
Michael> For things like SDP I think it makes sence to have UAR,
Michael> say, per process.
Michael> Looking at the non-SSE, 32 bit path, I would think you
Michael> want 3 spinlocks per UAR (for CQ, Send Q and Receive Q),
Michael> and not a global doorbell_lock. Granted, I dont know how
Michael> relevant any 32-bit non-SSE architectures are ...
Michael> I conclude the access layer needs some interface to
Michael> manage UARs. It might seem tavor specific but I think
Michael> something like UAR is what everyone has to do anyway to
Michael> have OS bypass, right?
Michael> Opinions?
Sounds interesting... I would be curious to see benchmarks for various
schemes. Exposing UARs to consumers seems a little bit of a layering
violation but if it turns out to be a big win then it might be worth
it. Of course I'm not sure how interested in optimizing 32-bit
architectures anyone is...
- R.
More information about the general
mailing list