[ofa-general] Re: failure to create an FMR mapping 1K pages on memfree

[Michael S. Tsirkin]

> Quoting Roland Dreier <rdreier at cisco.com>:
> Subject: Re: failure to create an FMR mapping 1K pages on memfree
> 
>  > > Is it really returning -ENOMEM?  It seems much more likely that you
>  > > are hitting the code
>  > > 
>  > > 	/* For Arbel, all MTTs must fit in the same page. */
>  > > 	if (mthca_is_memfree(dev) &&
>  > > 	    mr->attr.max_pages * sizeof *mr->mem.arbel.mtts > PAGE_SIZE)
>  > > 		return -EINVAL;
>  > > 
>  > > I guess you could call this limit a driver design issue.
>  > 
>  > Actually, I see this in fmr_pool.c:
>  > 
>  > 		fmr = kmalloc(sizeof *fmr + params->max_pages_per_fmr * sizeof (u64),
>  > 									GFP_KERNEL);
>  > 
>  > Therefore, for max_pages_per_fmr = 1K, this attempts to allocate 8K
>  > of physically contigious memory, which could explain the failure.
>  > 
>  > One way to fix this would be to use vmalloc to allocate this buffer.
>  > Opinions?
> 
> I don't think an order 1 allocation will fail under normal conditions.
> Larger allocations might fail, but I don't think vmalloc is the right
> solution... maybe just disable the caching in fmr_pool for larger
> FMRs?

Or just avoid FMR pool altogether.

> Anyway the issue here is defintely that mthca does not handle finding
> more than one page in the MTT table for memfree HCAs...

True.
On 64 bit systems, we could relatively easily solve this using vmap.
On 32 bit systems, it's a bigger change.


-- 
MST