[openib-general] [ANNOUNCE] GSI Implementation Candidate

[Grant Grundler]

On Mon, Aug 02, 2004 at 08:39:32AM -0700, Roland Dreier wrote:
> Right now what we do is a little ugly.  But we tell the HCA to create
> a memory region covering memory from address 0 up to (high_memory-PAGE_OFFSET)
> where address 0 is translated to address 0 (ie the identity mapping).

Is this representation of memory region using kernel virtual
or physical addresses?

> This works OK on the vast majority of systems but will break if we
> encounter systems with physical memory starting somewhere other than
> 0, discontiguous memory, a crazy PCI mapping that uses some other
> range of addresses, etc...

Most NUMA machines are very likely to break this assumption.
Even IA64 on ZX1 will break the assumptions about contigous memory.
THe physical memory map for ia64 is something like:
	0-1GB	RAM
	1-2	IOVA space (used by 32-bit PCI devices)
	2-4	MMIO space, CPUs, Firmware, etc
	...
	257-260 RAM (mem controller remapped this from 1-4GB space)

There's also space for more RAM and 64-bit MMIO (HP speak it's call GMMIO).

> In any case, yes you are right, the pci_map call should be on DMA-able
> memory (kmalloced memory or the like).

ok

> Once we have the basic verbs API all squared away, I would like to add
> an extension to get an L_Key with translation turned off.  This is
> almost like 'default L_Key' from the verbs extensions, but Tavor will
> require a protection domain as well.  This will let the HCA use any
> address returned from pci_map_xxx().

I don't understand how protection domains play with DMA. 
I gather so far the HCA hosts a virtual -> DMA mapping table
which the HCA driver keeps up to date. Is the protection domain
used by the HCA to limit which entries in the table a process on a
remote host may derefence?

grant