[ofa-general] Re: [ewg] [Patch mthca backport] Don't use kmalloc > 128k

[Jack Morgenstein]

On Thursday 16 July 2009 21:08, Doug Ledford wrote:
> On rhel4 and rhel5 machines, the kmalloc implementation does not  
> automatically forward kmalloc requests > 128kb to __get_free_pages.   
> Please include this patch in all rhel4 and rhel5 backport directories  
> so that we do the right thing in the mthca driver on rhel in regards  
> to kmalloc requests larger than 128k (at least in this code path,  
> there may be others lurking too, I'll forward additional patches if I  
> find they are needed).
> 
> 
commit a7f18a776785aecb5eb9967aef6f0f603b698ba0
Author: Doug Ledford <dledford at redhat.com>
Date:   Thu Jul 16 12:47:55 2009 -0400

    [mthca] Fix attempts to use kmalloc on overly large allocations
    
    Signed-off-by: Doug Ledford <dledford at redhat.com>

----

Roland,
I think that this patch should be taken into the mainstream kernel, rather
than just as a backport patch for RHEL.  (We can have a similar patch for mlx4).
I notice that __get_free_pages(), free_pages(), and get_order() are all in the
mainstream kernel.

This will fix the 2^20 bits limit on our bitmaps once and for all.
If you agree, I will post this patch and one for mlx4 on the general list.

Doug posted this patch on the EWG list.

Thanks Doug!

diff --git a/drivers/infiniband/hw/mthca/mthca_mr.c b/drivers/infiniband/hw/mthca/mthca_mr.c
index d606edf..312e18d 100644
--- a/drivers/infiniband/hw/mthca/mthca_mr.c
+++ b/drivers/infiniband/hw/mthca/mthca_mr.c
@@ -152,8 +152,11 @@ static int mthca_buddy_init(struct mthca_buddy *buddy, int max_order)
 		goto err_out;
 
 	for (i = 0; i <= buddy->max_order; ++i) {
-		s = BITS_TO_LONGS(1 << (buddy->max_order - i));
-		buddy->bits[i] = kmalloc(s * sizeof (long), GFP_KERNEL);
+		s = BITS_TO_LONGS(1 << (buddy->max_order - i)) * sizeof(long);
+		if(s > PAGE_SIZE)
+			buddy->bits[i] = (unsigned long *)__get_free_pages(GFP_KERNEL, get_order(s));
+		else 
+			buddy->bits[i] = kmalloc(s, GFP_KERNEL);
 		if (!buddy->bits[i])
 			goto err_out_free;
 		bitmap_zero(buddy->bits[i],
@@ -166,9 +169,13 @@ static int mthca_buddy_init(struct mthca_buddy *buddy, int max_order)
 	return 0;
 
 err_out_free:
-	for (i = 0; i <= buddy->max_order; ++i)
-		kfree(buddy->bits[i]);
-
+	for (i = 0; i <= buddy->max_order; ++i){
+		s = BITS_TO_LONGS(1 << (buddy->max_order - i)) * sizeof(long);
+		if(s > PAGE_SIZE)
+			free_pages((unsigned long)buddy->bits[i], get_order(s));
+		else
+			kfree(buddy->bits[i]);
+	}
 err_out:
 	kfree(buddy->bits);
 	kfree(buddy->num_free);
@@ -178,10 +185,15 @@ err_out:
 
 static void mthca_buddy_cleanup(struct mthca_buddy *buddy)
 {
-	int i;
+	int i, s;
 
-	for (i = 0; i <= buddy->max_order; ++i)
-		kfree(buddy->bits[i]);
+	for (i = 0; i <= buddy->max_order; ++i){
+		s = BITS_TO_LONGS(1 << (buddy->max_order - i)) * sizeof(long);
+		if(s > PAGE_SIZE)
+			free_pages((unsigned long)buddy->bits[i], get_order(s));
+		else
+			kfree(buddy->bits[i]);
+	}
 
 	kfree(buddy->bits);
 	kfree(buddy->num_free);