[openib-general] Reserved L_Key API

[Grant Grundler]

On Sat, Sep 18, 2004 at 01:47:18PM -0700, Roland Dreier wrote:
> The device knows what its DMA mask is

It does?
The driver knows. But doesn't have to tell the device anything
if the mask happens to be smaller than number of physical
bits the device can use for DMA.

> so there's no problem
> if an architecture limits it to DMA only in the low 4G.

It's the chipset that limits the DMA, not the arch.
e.g. if a PCI Host bus controller doesn't support DAC (Dual address
cycle), then the PCI device can only use 32-bit DMA. DMA
mapping services or higher level OS code has to make sure 
we never hand back a DMA address > 32-bits.

...
> For systems with an IOMMU, the driver can just set its DMA
> mask to the low 4G and be happy.  For systems without IOMMU (ie Intel
> IA32/EMT64) the driver can set its DMA mask to the full 64 bits and
> create an MR spanning the low 64G and still be happy.

The world can mostly be divided between with and without IOMMU.
But it's not a real clean division given some exceptions.
Let me list two that are well known:
o ZX1 and SX1000 chipsets (HP): ZX1 chipset can allow 64-bit
  devices to bypass the iommu.  DMA mapping services just needs
  to know if the device is 64-bit capable and then it will
  allow that. ZX1 and SX1000 are used for both parisc (PA8800)
  and IA64.

o "EMT64" (AMD64): Has a GART which is also used as an IOMMU
  by 32-bit PCI devices.  64-bit devices can DMA directly into RAM.
  But the DMA services are involved at setup time.
  There's probably a bit more to it.

o SGI Altix (IA64) also has an IOMMU but has different requirements
  for PCI-X than PCI devices.  It allows "64-bit" DMA but then
  uses some of the upper bits as "hints" to the chipset.

The IB code should only need to know what the largest acceptable
DMA mask is and then it can build a window based on that mask.
Or only attempt to build a "global" window if all 64-bits are
"available" for DMA. As noted above, that may not work on SGI Altix
but I suspect other major problems (broken PCI ordering rules) 
will make IB hard to get working properly there.

hth,
grant