[openib-general] basic IB doubt

[Asgeir Eiriksson]

> Date: Mon, 28 Aug 2006 10:22:52 -0600> From: jgunthorpe at obsidianresearch.com> To: Thomas.Talpey at netapp.com> CC: glebn at voltaire.com; openib-general at openib.org> Subject: Re: [openib-general] basic IB doubt> > On Mon, Aug 28, 2006 at 10:38:43AM -0400, Talpey, Thomas wrote:> > > Will turning on the Opteron's IOMMU introduce some of these> > issues to x86?> > No, definately not. The Opteron IOMMU (the GART) is a pure address> translation mechanism and doesn't change the operation of the caches.> > If Sun has a problem on larger systems I wonder if SGI Altix also has a> problem? SGI Altix is definately a real system that people use IB> cards in today and it would be easy to imagine such a large system> could have coherence issues with memory polling..> 
Jason
 
Yes, there's an issue with polling on the last byte of data, but not polling on a properly implemented CQ.
 
The SGI machines have fully cache coherent I/O, e.g. as part of a DMA write an I/O controller will invalidate
any cached copies of a cache line being DMA written. There is still an ordering issue with respect to polling on
the last byte of data. For example assume an incoming RDMA WRITE results in two DMA writes to two different
pages, then the ordering of these DMA write vis-a-vis being visible in the coherent domain is not guaranteed.
Instead these machines have an I/O sync operation to implement ordering guarantees, e.g. when such an I/O
sync is used in our example to implement the CQ then it's guaranteed that both DMA writes have completed
vis-a-vis being visible in the coherent domain.
 
Hope that helps,
 
Asgeir Eiriksson
CTO
Chelsio Communications, Inc.
_________________________________________________________________
Express yourself with gadgets on Windows Live Spaces
http://discoverspaces.live.com?source=hmtag1&loc=us
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20060828/e1e90bcf/attachment.html>