[ofa-general] Re: failure to create an FMR mapping 1K pages on memfree

[Roland Dreier]

 > > Is it really returning -ENOMEM?  It seems much more likely that you
 > > are hitting the code
 > > 
 > > 	/* For Arbel, all MTTs must fit in the same page. */
 > > 	if (mthca_is_memfree(dev) &&
 > > 	    mr->attr.max_pages * sizeof *mr->mem.arbel.mtts > PAGE_SIZE)
 > > 		return -EINVAL;
 > > 
 > > I guess you could call this limit a driver design issue.
 > 
 > Actually, I see this in fmr_pool.c:
 > 
 > 		fmr = kmalloc(sizeof *fmr + params->max_pages_per_fmr * sizeof (u64),
 > 									GFP_KERNEL);
 > 
 > Therefore, for max_pages_per_fmr = 1K, this attempts to allocate 8K
 > of physically contigious memory, which could explain the failure.
 > 
 > One way to fix this would be to use vmalloc to allocate this buffer.
 > Opinions?

I don't think an order 1 allocation will fail under normal conditions.
Larger allocations might fail, but I don't think vmalloc is the right
solution... maybe just disable the caching in fmr_pool for larger
FMRs?

Anyway the issue here is defintely that mthca does not handle finding
more than one page in the MTT table for memfree HCAs...