<html>
<head>
<style>
P
{
margin:0px;
padding:0px
}
body
{
FONT-SIZE: 10pt;
FONT-FAMILY:Tahoma
}
</style>
</head>
<body><BR><BR><BR><BR><BR>
<HR id=stopSpelling>
> Date: Mon, 28 Aug 2006 10:22:52 -0600<BR>> From: jgunthorpe@obsidianresearch.com<BR>> To: Thomas.Talpey@netapp.com<BR>> CC: glebn@voltaire.com; openib-general@openib.org<BR>> Subject: Re: [openib-general] basic IB doubt<BR>> <BR>> On Mon, Aug 28, 2006 at 10:38:43AM -0400, Talpey, Thomas wrote:<BR>> <BR>> > Will turning on the Opteron's IOMMU introduce some of these<BR>> > issues to x86?<BR>> <BR>> No, definately not. The Opteron IOMMU (the GART) is a pure address<BR>> translation mechanism and doesn't change the operation of the caches.<BR>> <BR>> If Sun has a problem on larger systems I wonder if SGI Altix also has a<BR>> problem? SGI Altix is definately a real system that people use IB<BR>> cards in today and it would be easy to imagine such a large system<BR>> could have coherence issues with memory polling..<BR>> <BR><BR>
Jason<BR>
<BR>
Yes, there's an issue with polling on the last byte of data, but not polling on a properly implemented CQ.<BR>
<BR>
The SGI machines have fully cache coherent I/O, e.g. as part of a DMA write an I/O controller will invalidate<BR>
any cached copies of a cache line being DMA written. There is still an ordering issue with respect to polling on<BR>
the last byte of data. For example assume an incoming RDMA WRITE results in two DMA writes to two different<BR>
pages, then the ordering of these DMA write vis-a-vis being visible in the coherent domain is not guaranteed.<BR>
Instead these machines have an I/O sync operation to implement ordering guarantees, e.g. when such an I/O<BR>
sync is used in our example to implement the CQ then it's guaranteed that both DMA writes have completed<BR>
vis-a-vis being visible in the coherent domain.<BR>
<BR>
Hope that helps,<BR>
<BR>
Asgeir Eiriksson<BR>
CTO<BR>
Chelsio Communications, Inc.<BR><br /><hr />Express yourself with gadgets on Windows Live Spaces <a href='http://discoverspaces.live.com?source=hmtag1&loc=us' target='_new'>Try it!</a></body>
</html>