[openib-general] [PATCH] IB/documentation - add new file to Documentation/infiniband

[James Lentini]

Ralph,

Thanks for writing this! It will be a big help to ULP authors.

A few suggestions below:

On Thu, 9 Nov 2006, Ralph Campbell wrote:

> This patch adds a new file to the kernel infiniband documentation
> directory to briefly describe how to use memory regions.
> 
> Note: I will be on vacation from Nov. 11 through Nov. 26.
> 
> Signed-off-by: Ralph Campbell <ralph.campbell at qlogic.com>
> 
> diff -r b9d92097f918 Documentation/infiniband/memory_regions.txt
> --- /dev/null	Thu Jan 01 00:00:00 1970 +0000
> +++ b/Documentation/infiniband/memory_regions.txt	Wed Nov 08 18:35:46 2006 -0800
> @@ -0,0 +1,110 @@
<snip>
> +Kernel Space Memory Regions
> +
> +  ib_get_dma_mr()  This function returns a pointer to struct ib_mr
                                                        ^
                                                        a

> +  which contains the 'lkey' and 'rkey' fields similar to user
> +  memory regions.  The memory region represents all of physical
> +  memory so no base address or length is needed when creating it.
> +  The addresses used for the 'addr' field of struct ib_sge need
> +  to be hardware device addresses suitable for DMA.
                                                     ^
                                          access by RDMA devices.

> +  Since this mapping may be device specific, there are a set
> +  of kernel verbs functions corresponding to the DMA mapping
> +  functions described in DMA-API.txt.  Another useful reference
> +  is the "Linux Device Drivers" book, 3rd edition, by Rubini and Corbet.
> +
> +	ib_dma_mapping_error()
> +	ib_dma_map_single()
> +	ib_dma_unmap_single()
> +	ib_dma_map_page()
> +	ib_dma_unmap_page()
> +	ib_dma_map_sg()
> +	ib_dma_unmap_sg()
> +	ib_sg_dma_address()
> +	ib_sg_dma_len()
> +	ib_dma_sync_single_for_cpu()
> +	ib_dma_sync_single_for_device()
> +
> +  Remote processes should use the same address for 'remote_addr'
> +  as the local kernel's address as returned by the mapping functions
> +  listed above.  The only difference is the local kernel uses the
> +  'lkey' and the remote kernel uses the 'rkey'.

I found the above paragraph difficult to understand on the first 
read. How about structuring the text similar to the user space 
explanation above:

 The addresses returned by these mapping functions should be 
 used for a struct ib_send_wr's 'remote_addr' field with the 
 appropriate rkey.

> +  Note that the mapped addresses need to be unmapped after they
> +  are no longer needed.  This may require the local and remote
> +  kernels to pass messages at the middle or upper layers to
> +  sychronize.
> +
> +  ib_reg_phys_mr()  This function returns a pointer to struct ib_mr.
> +  It takes an array of device DMA addresses and lengths which are used
> +  to describe the memory region.  These addresses are created by
> +  calling the mapping functions listed for ib_get_dma_mr().

The iova is an in/out parameter. I recommend including a description 
of how to initialize it:

 The 'iova' argument can be used by the caller to request an address 
 to associate with the first byte of the address region. Upon return, 
 the 'iova' argument is the ... <continue with below>

> +  The 'iova' argument is the starting address of the memory region
> +  which should be used with the 'lkey' or 'rkey' returned in the
> +  struct ib_mr.
> +
> +  ib_dereg_mr() is used to destroy memory regions created by
> +  either ib_get_dma_mr() or ib_reg_phys_mr().
> +
> +  ib_alloc_fmr()  This returns a pointer to a struct ib_fmr.
> +  The struct ib_fmr_attr argument specifies the size of each
> +  FMR "page" as a power of two in 'page_shift'.  This size
> +  is assumed by ib_map_phys_fmr() described below.
> +  A FMR cannot be used until ib_map_phys_fmr() is called.
> +  The 'lkey' and 'rkey' fields are defined in struct ib_fmr
> +  and used the same way as the other memory regions.
> +
> +  ib_map_phys_fmr()  The function takes an array of u64 and a length
> +  for the number of entries in the array.  Each u64 value should be
> +  a DMA address created with the mapping functions listed for
> +  ib_get_dma_mr().  The length of each u64 address region is the
> +  FMR page size set when ib_alloc_fmr() was called.
> +  Note that this now defines the memory region to start at address
> +  'iova' and is the base address used for 'addr' and 'remote_addr'.
> +  The size of the memory region is the array length times the
> +  FMR page size.
> +
> +  FMR memory regions should be unmapped by calling ib_unmap_fmr()
> +  and then the ib_fmr destroyed by calling ib_dealloc_fmr().
> +
> +  See also ib_create_fmr_pool(), ib_fmr_pool_map_phys(), and
> +  ib_fmr_pool_unmap() which are defined in the ib_core module
> +  to assist in caching FMRs.  This can help performance when
> +  the same memory is mapped/unmapped frequently.
> +
> +  Despite the name FMR, the memory region allocation and deallocation
> +  functions perform very differently depending on device, processor,
> +  and platform differences.
> 
> 
> 
> _______________________________________________
> openib-general mailing list
> openib-general at openib.org
> http://openib.org/mailman/listinfo/openib-general
> 
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
>