[ewg] Re: [PATCH v3] mlx4_ib: Optimize hugetlab pages support

[Eli Cohen]

On Mon, Apr 13, 2009 at 10:19:08PM +0300, Yossi Etigin wrote:

I see. That surprises me... could you also print vma->vm_start and
vma->vm_end? I would have expected the vma to be streched to fill a
full huge page and return the number of regular pages fitting in it.

> Eli Cohen wrote:
> > On Mon, Apr 06, 2009 at 08:49:43PM +0300, Yossi Etigin wrote:
> >> I don't understand - if all area is huge pages, it does not mean that
> >> it fills full huge pages - I can have just 4096 bytes in huge page memory 
> >> and umem->hugetlb will remain 1, right?
> > 
> > You may call ib_umem_get() with a fraction of a huge page but I expect
> > the number of pages returned from get_user_pages() will fill up a huge
> > page. Can you check that with the mckey test you were using?
> 
> The number of pages is 1.
> I got this in dmesg with the modified mckey (see the last line):
> 
> umem: addr=508000 size=1024 hugetlb=0 npages=1
> umem: addr=50a000 size=4096 hugetlb=0 npages=1
> umem: addr=50c000 size=4352 hugetlb=0 npages=2
> umem: addr=50f000 size=4096 hugetlb=0 npages=1
> umem: addr=2aaaaac00000 size=140 hugetlb=1 npages=1
> 
> 
> After applying this to umem.c:
> 
> --- ofa_kernel-1.4.1/drivers/infiniband/core/umem.c	2009-04-13 22:15:19.000000000 +0300
> +++ ofa_kernel-1.4.1.patched/drivers/infiniband/core/umem.c	2009-04-13 22:09:36.000000000 +0300
> @@ -137,6 +137,7 @@
>  	int ret;
>  	int off;
>  	int i;
> +	int ntotalpages;
>  	DEFINE_DMA_ATTRS(attrs);
>  
>  	if (dmasync)
> @@ -196,6 +197,7 @@
>  	cur_base = addr & PAGE_MASK;
>  
>  	ret = 0;
> +	ntotalpages = 0;
>  	while (npages) {
>  		ret = get_user_pages(current, current->mm, cur_base,
>  				     min_t(unsigned long, npages,
> @@ -226,6 +228,7 @@
>  				    !is_vm_hugetlb_page(vma_list[i + off]))
>  					umem->hugetlb = 0;
>  				sg_set_page(&chunk->page_list[i], page_list[i + off], PAGE_SIZE, 0);
> +				ntotalpages++;
>  			}
>  
>  			chunk->nmap = ib_dma_map_sg_attrs(context->device,
> @@ -254,8 +257,11 @@
>  	if (ret < 0) {
>  		__ib_umem_release(context->device, umem, 0);
>  		kfree(umem);
> -	} else
> +	} else {
>  		current->mm->locked_vm = locked;
> +	 	printk(KERN_DEBUG "umem: addr=%lx size=%ld hugetlb=%d npages=%d\n",
> +		       addr, size, umem->hugetlb, ntotalpages);
> +	}
>  
>  	up_write(&current->mm->mmap_sem);
>  	if (vma_list)