[ofa-general] Re: failure to create an FMR mapping 1K pages on memfree

[Michael S. Tsirkin]

> Quoting Roland Dreier <rdreier at cisco.com>:
> Subject: Re: failure to create an FMR mapping 1K pages on memfree
> 
>  > I have got a report on failure to create FMR mapping 1K pages (that is
>  > 4MB) on memfree.
>  >
>  > I don't have the exact details (ie if Arbel/Sinai / what FW  / etc)
>  > nor which exact check fails in
>  > mthca_fmr_alloc, but what's clear is that the latter function returns
>  > -ENOMEM when attr.max_pages is 1024 and it works fine when
>  > attr.max_pages is 256.
>  >
>  > Is this failure clear to you? if yes, does a HW or FW limit is being
>  > hit or its a driver design issue?
> 
> Is it really returning -ENOMEM?  It seems much more likely that you
> are hitting the code
> 
> 	/* For Arbel, all MTTs must fit in the same page. */
> 	if (mthca_is_memfree(dev) &&
> 	    mr->attr.max_pages * sizeof *mr->mem.arbel.mtts > PAGE_SIZE)
> 		return -EINVAL;
> 
> I guess you could call this limit a driver design issue.

Actually, I see this in fmr_pool.c:

		fmr = kmalloc(sizeof *fmr + params->max_pages_per_fmr * sizeof (u64),
									GFP_KERNEL);

Therefore, for max_pages_per_fmr = 1K, this attempts to allocate 8K
of physically contigious memory, which could explain the failure.

One way to fix this would be to use vmalloc to allocate this buffer.
Opinions?

-- 
MST