[openib-general] [PATCH] IB/documentation - add new file to Documentation/infiniband

[Ralph Campbell]

On Fri, 2006-11-10 at 10:34 -0500, James Lentini wrote:
> Ralph,
> 
> Thanks for writing this! It will be a big help to ULP authors.

You are welcome.

> A few suggestions below:
> 
> On Thu, 9 Nov 2006, Ralph Campbell wrote:
> 
> > This patch adds a new file to the kernel infiniband documentation
> > directory to briefly describe how to use memory regions.
> > 
> > Note: I will be on vacation from Nov. 11 through Nov. 26.
> > 
> > Signed-off-by: Ralph Campbell <ralph.campbell at qlogic.com>
> > 
> > diff -r b9d92097f918 Documentation/infiniband/memory_regions.txt
> > --- /dev/null	Thu Jan 01 00:00:00 1970 +0000
> > +++ b/Documentation/infiniband/memory_regions.txt	Wed Nov 08 18:35:46 2006 -0800
> > @@ -0,0 +1,110 @@
> <snip>
> > +Kernel Space Memory Regions
> > +
> > +  ib_get_dma_mr()  This function returns a pointer to struct ib_mr
>                                                         ^
>                                                         a

OK.

> > +  which contains the 'lkey' and 'rkey' fields similar to user
> > +  memory regions.  The memory region represents all of physical
> > +  memory so no base address or length is needed when creating it.
> > +  The addresses used for the 'addr' field of struct ib_sge need
> > +  to be hardware device addresses suitable for DMA.
>                                                      ^
>                                           access by RDMA devices.

OK.

> > +  Since this mapping may be device specific, there are a set
> > +  of kernel verbs functions corresponding to the DMA mapping
> > +  functions described in DMA-API.txt.  Another useful reference
> > +  is the "Linux Device Drivers" book, 3rd edition, by Rubini and Corbet.
> > +
> > +	ib_dma_mapping_error()
> > +	ib_dma_map_single()
> > +	ib_dma_unmap_single()
> > +	ib_dma_map_page()
> > +	ib_dma_unmap_page()
> > +	ib_dma_map_sg()
> > +	ib_dma_unmap_sg()
> > +	ib_sg_dma_address()
> > +	ib_sg_dma_len()
> > +	ib_dma_sync_single_for_cpu()
> > +	ib_dma_sync_single_for_device()
> > +
> > +  Remote processes should use the same address for 'remote_addr'
> > +  as the local kernel's address as returned by the mapping functions
> > +  listed above.  The only difference is the local kernel uses the
> > +  'lkey' and the remote kernel uses the 'rkey'.
> 
> I found the above paragraph difficult to understand on the first 
> read. How about structuring the text similar to the user space 
> explanation above:
> 
>  The addresses returned by these mapping functions should be 
>  used for a struct ib_send_wr's 'remote_addr' field with the 
>  appropriate rkey.

OK.
I guess I was trying to point out that the address used with the
lkey or rkey will always be the same when accessing the same byte
of memory. It is just that the lkey is used when accessing memory
locally and the rkey is used to access the memory remotely.

> > +  Note that the mapped addresses need to be unmapped after they
> > +  are no longer needed.  This may require the local and remote
> > +  kernels to pass messages at the middle or upper layers to
> > +  sychronize.
> > +
> > +  ib_reg_phys_mr()  This function returns a pointer to struct ib_mr.
> > +  It takes an array of device DMA addresses and lengths which are used
> > +  to describe the memory region.  These addresses are created by
> > +  calling the mapping functions listed for ib_get_dma_mr().
> 
> The iova is an in/out parameter. I recommend including a description 
> of how to initialize it:
> 
>  The 'iova' argument can be used by the caller to request an address 
>  to associate with the first byte of the address region. Upon return, 
>  the 'iova' argument is the ... <continue with below>

Iova was at one point in time an in/out parameter but was changed
to be just an in parameter some time ago.

I modified the text as follows:

The 'iova' argument can be used by the caller to set the address
of the first byte of the address region.  The addresses used
with struct ib_sge 'addr' and struct ib_send_wr 'remote_addr' are
thus 'iova' plus offset within the length of the memory region.
The 'lkey' and 'rkey' fields for the above structs should be set
with the values returned in the struct ib_mr 'lkey' and 'rkey' fields.

I will update the patch and resend it with these changes.