[ewg] Re: [PATCH v3] mlx4_ib: Optimize hugetlab pages support

[Yossi Etigin]

Eli Cohen wrote:
> On Mon, Apr 06, 2009 at 08:49:43PM +0300, Yossi Etigin wrote:
>> I don't understand - if all area is huge pages, it does not mean that
>> it fills full huge pages - I can have just 4096 bytes in huge page memory 
>> and umem->hugetlb will remain 1, right?
> 
> You may call ib_umem_get() with a fraction of a huge page but I expect
> the number of pages returned from get_user_pages() will fill up a huge
> page. Can you check that with the mckey test you were using?

The number of pages is 1.
I got this in dmesg with the modified mckey (see the last line):

umem: addr=508000 size=1024 hugetlb=0 npages=1
umem: addr=50a000 size=4096 hugetlb=0 npages=1
umem: addr=50c000 size=4352 hugetlb=0 npages=2
umem: addr=50f000 size=4096 hugetlb=0 npages=1
umem: addr=2aaaaac00000 size=140 hugetlb=1 npages=1


After applying this to umem.c:

--- ofa_kernel-1.4.1/drivers/infiniband/core/umem.c	2009-04-13 22:15:19.000000000 +0300
+++ ofa_kernel-1.4.1.patched/drivers/infiniband/core/umem.c	2009-04-13 22:09:36.000000000 +0300
@@ -137,6 +137,7 @@
 	int ret;
 	int off;
 	int i;
+	int ntotalpages;
 	DEFINE_DMA_ATTRS(attrs);
 
 	if (dmasync)
@@ -196,6 +197,7 @@
 	cur_base = addr & PAGE_MASK;
 
 	ret = 0;
+	ntotalpages = 0;
 	while (npages) {
 		ret = get_user_pages(current, current->mm, cur_base,
 				     min_t(unsigned long, npages,
@@ -226,6 +228,7 @@
 				    !is_vm_hugetlb_page(vma_list[i + off]))
 					umem->hugetlb = 0;
 				sg_set_page(&chunk->page_list[i], page_list[i + off], PAGE_SIZE, 0);
+				ntotalpages++;
 			}
 
 			chunk->nmap = ib_dma_map_sg_attrs(context->device,
@@ -254,8 +257,11 @@
 	if (ret < 0) {
 		__ib_umem_release(context->device, umem, 0);
 		kfree(umem);
-	} else
+	} else {
 		current->mm->locked_vm = locked;
+	 	printk(KERN_DEBUG "umem: addr=%lx size=%ld hugetlb=%d npages=%d\n",
+		       addr, size, umem->hugetlb, ntotalpages);
+	}
 
 	up_write(&current->mm->mmap_sem);
 	if (vma_list)