[ofa-general] [PATCH RFC] RDMA: New Memory Extensions.

[Ralph Campbell]

On Wed, 2008-05-14 at 19:49 -0700, Roland Dreier wrote:
>  > So you want the page size specified in the fast_reg_page_list as
>  > opposed to when the page list is bound to the fast_reg mr (via
>  > post_send)?
> 
> It's kind of the same thing, since the fast_reg_page_list is part of the
> send work request... the structures you have at the moment are:
> 
>  > +		struct {
>  > +			u64				iova_start;
>  > +			struct ib_fast_reg_page_list	*page_list;
>  > +			int				fbo;
>  > +			u32				length;
>  > +			int				access_flags;
>  > +			struct ib_mr 			*mr;
> 
> (side note... move this pointer up with the other pointers, so you don't
> end up with a hole in the structure due to alignment... or stick an int
> page_size in to fill the hole)
> 
>  > +		} fast_reg;
> 
>  > +struct ib_fast_reg_page_list {
>  > +	struct ib_device 	*device;
>  > +	u64			*page_list;
>  > +	int			page_list_len;
>  > +};
> 
> is page_list_len the maximum length of the page_list, or is it filled in
> by the consumer?  The driver could figure out the length of the
> page_list for any given work request by looking at the MR length and the
> page_size I suppose.
> 
>  - R.

I think Roland and Steve misunderstood what I was asking about
the struct ib_fast_reg_page_list * returned from
ib_alloc_fast_reg_page_list().

The question is "what can the caller do with the pointer?"
Clearly, the caller can pass the pointer to
ib_post_send(IB_WR_FAST_REG_MR) and use the [LR]_Key in the
normal ways.

Can the caller dereference the pointer and look at the
values in page_list[]? Are these values understood to be
a physical addresses that can be passed to phys_to_virt() for example?
Are they byte addresses always aligned to a page boundary?

The reason I ask is that the address used with the [LR]_Key from
ib_get_dma_mr() has to be translated with ib_dma_map_single(), etc.
because the ipath driver doesn't necessarily use physical addresses
for the address in the send WQEs. Normally, the address in the
send WQE is a kernel virtual address so the ib_ipath driver can
memcpy() the data to the chip.

Lets say that ib_ipath uses vmalloc() to allocate the pages
instead of dma_alloc_coherent(). As long as the ULP only uses
the page_list values as an uninterpreted number that is passed
back to the driver via subsequent verbs calls, it wouldn't
matter to the ULP what the number represents. But if the ULP
expects to be able to call some other kernel function to
map or translate that value, then the ULP has to know what
kind of number it represents, its size and alignment, etc.