[openib-general] [PATCH] IB/documentation - add new file to Documentation/infiniband

[Ralph Campbell]

This patch adds a new file to the kernel infiniband documentation
directory to briefly describe how to use memory regions.

Note: I will be on vacation from Nov. 11 through Nov. 26.

Signed-off-by: Ralph Campbell <ralph.campbell at qlogic.com>

diff -r b9d92097f918 Documentation/infiniband/memory_regions.txt
--- /dev/null	Thu Jan 01 00:00:00 1970 +0000
+++ b/Documentation/infiniband/memory_regions.txt	Wed Nov 08 18:35:46 2006 -0800
@@ -0,0 +1,110 @@
+INFINIBAND MEMORY REGIONS
+
+  This is an overview of memory region usage for the user and kernel
+  verbs interface.  The verbs API to send and receive data does not
+  specify memory addresses directly.  Instead, a memory region
+  is constructed and a Lkey or Rkey is used to refer to the region.
+
+User Space Memory Regions
+
+  User space memory regions are created by calling ibv_reg_mr().
+  It returns a pointer to a struct ibv_mr which contains the
+  'lkey' field and 'rkey' field.  The lkey should be copied
+  into the 'lkey' field of struct ibv_sge when posting buffers
+  with ibv_post_send(), ibv_post_recv(), and ibv_post_srq_recv().
+  The 'addr' field of the ibv_sge should be a user address between
+  the address and address + length passed to ibv_reg_mr().
+
+  The 'rkey' can be sent to another process and used by the
+  remote process in RDMA write, read, and atomic operations
+  to access the local process' memory region.
+  The 'remote_addr' field in the ibv_send_wr should be the local
+  process' address within the memory region.  At some point in
+  the future, the interface may be extended to allow zero based
+  remote addresses which would mean the remote_addr would be
+  an offset within the local process' memory region.
+
+  A memory region is destroyed by calling ibv_dereg_mr().
+
+  Note that creating and destroying memory regions results
+  in kernel system calls which lock the user's virtual memory
+  to physical memory.  This means the system administrator must set
+  the RLIMIT memory lock limit high enough for processes to
+  be able to create memory regions of the desired size.
+  It is therefore best to limit the size of memory regions created.
+
+Kernel Space Memory Regions
+
+  ib_get_dma_mr()  This function returns a pointer to struct ib_mr
+  which contains the 'lkey' and 'rkey' fields similar to user
+  memory regions.  The memory region represents all of physical
+  memory so no base address or length is needed when creating it.
+  The addresses used for the 'addr' field of struct ib_sge need
+  to be hardware device addresses suitable for DMA.
+  Since this mapping may be device specific, there are a set
+  of kernel verbs functions corresponding to the DMA mapping
+  functions described in DMA-API.txt.  Another useful reference
+  is the "Linux Device Drivers" book, 3rd edition, by Rubini and Corbet.
+
+	ib_dma_mapping_error()
+	ib_dma_map_single()
+	ib_dma_unmap_single()
+	ib_dma_map_page()
+	ib_dma_unmap_page()
+	ib_dma_map_sg()
+	ib_dma_unmap_sg()
+	ib_sg_dma_address()
+	ib_sg_dma_len()
+	ib_dma_sync_single_for_cpu()
+	ib_dma_sync_single_for_device()
+
+  Remote processes should use the same address for 'remote_addr'
+  as the local kernel's address as returned by the mapping functions
+  listed above.  The only difference is the local kernel uses the
+  'lkey' and the remote kernel uses the 'rkey'.
+
+  Note that the mapped addresses need to be unmapped after they
+  are no longer needed.  This may require the local and remote
+  kernels to pass messages at the middle or upper layers to
+  sychronize.
+
+  ib_reg_phys_mr()  This function returns a pointer to struct ib_mr.
+  It takes an array of device DMA addresses and lengths which are used
+  to describe the memory region.  These addresses are created by
+  calling the mapping functions listed for ib_get_dma_mr().
+  The 'iova' argument is the starting address of the memory region
+  which should be used with the 'lkey' or 'rkey' returned in the
+  struct ib_mr.
+
+  ib_dereg_mr() is used to destroy memory regions created by
+  either ib_get_dma_mr() or ib_reg_phys_mr().
+
+  ib_alloc_fmr()  This returns a pointer to a struct ib_fmr.
+  The struct ib_fmr_attr argument specifies the size of each
+  FMR "page" as a power of two in 'page_shift'.  This size
+  is assumed by ib_map_phys_fmr() described below.
+  A FMR cannot be used until ib_map_phys_fmr() is called.
+  The 'lkey' and 'rkey' fields are defined in struct ib_fmr
+  and used the same way as the other memory regions.
+
+  ib_map_phys_fmr()  The function takes an array of u64 and a length
+  for the number of entries in the array.  Each u64 value should be
+  a DMA address created with the mapping functions listed for
+  ib_get_dma_mr().  The length of each u64 address region is the
+  FMR page size set when ib_alloc_fmr() was called.
+  Note that this now defines the memory region to start at address
+  'iova' and is the base address used for 'addr' and 'remote_addr'.
+  The size of the memory region is the array length times the
+  FMR page size.
+
+  FMR memory regions should be unmapped by calling ib_unmap_fmr()
+  and then the ib_fmr destroyed by calling ib_dealloc_fmr().
+
+  See also ib_create_fmr_pool(), ib_fmr_pool_map_phys(), and
+  ib_fmr_pool_unmap() which are defined in the ib_core module
+  to assist in caching FMRs.  This can help performance when
+  the same memory is mapped/unmapped frequently.
+
+  Despite the name FMR, the memory region allocation and deallocation
+  functions perform very differently depending on device, processor,
+  and platform differences.