[ewg] [PATCH 1/2] fmr_pool flush serials can get out of sync

[Olaf Kirch]

From: Olaf Kirch <olaf.kirch at oracle.com>
Subject: [fmr_pool] fmr_pool flush serials can get out of sync

Normally, the serial numbers for flush requests and flushes
executed should be in sync.

When we decide to flush dirty MRs because there are too many of them, we
wake up the cleanup thread and let it do its stuff.  As a side effect, it
increments pool->flush_ser, which leaves it one higher than req_ser. The
next time the user calls ib_flush_fmr_pool, it will wake up the cleanup
thread, but won't wait for the flush to complete. This can cause memory
corruption, as the user expects the flush to have taken place.

Signed-off-by: Olaf Kirch <olaf.kirch at oracle.com>
---
 drivers/infiniband/core/fmr_pool.c |   16 ++++++++++------
 1 file changed, 10 insertions(+), 6 deletions(-)

Index: ofa_kernel-1.3/drivers/infiniband/core/fmr_pool.c
===================================================================
--- ofa_kernel-1.3.orig/drivers/infiniband/core/fmr_pool.c
+++ ofa_kernel-1.3/drivers/infiniband/core/fmr_pool.c
@@ -182,8 +182,7 @@ static int ib_fmr_cleanup_thread(void *p
 	struct ib_fmr_pool *pool = pool_ptr;
 
 	do {
-		if (pool->dirty_len >= pool->dirty_watermark ||
-		    atomic_read(&pool->flush_ser) - atomic_read(&pool->req_ser) < 0) {
+		if (atomic_read(&pool->flush_ser) - atomic_read(&pool->req_ser) < 0) {
 			ib_fmr_batch_release(pool);
 
 			atomic_inc(&pool->flush_ser);
@@ -194,8 +193,7 @@ static int ib_fmr_cleanup_thread(void *p
 		}
 
 		set_current_state(TASK_INTERRUPTIBLE);
-		if (pool->dirty_len < pool->dirty_watermark &&
-		    atomic_read(&pool->flush_ser) - atomic_read(&pool->req_ser) >= 0 &&
+		if (atomic_read(&pool->flush_ser) - atomic_read(&pool->req_ser) >= 0 &&
 		    !kthread_should_stop())
 			schedule();
 		__set_current_state(TASK_RUNNING);
@@ -397,6 +395,7 @@ EXPORT_SYMBOL(ib_destroy_fmr_pool);
  */
 int ib_flush_fmr_pool(struct ib_fmr_pool *pool)
 {
+	int flush_count = atomic_read(&pool->flush_ser);
 	int serial = atomic_inc_return(&pool->req_ser);
 
 	wake_up_process(pool->thread);
@@ -405,6 +404,9 @@ int ib_flush_fmr_pool(struct ib_fmr_pool
 				     atomic_read(&pool->flush_ser) - serial >= 0))
 		return -EINTR;
 
+	flush_count = atomic_read(&pool->flush_ser) - flush_count;
+	BUG_ON(flush_count == 0);
+
 	return 0;
 }
 EXPORT_SYMBOL(ib_flush_fmr_pool);
@@ -511,8 +513,10 @@ int ib_fmr_pool_unmap(struct ib_pool_fmr
 			list_add_tail(&fmr->list, &pool->free_list);
 		} else {
 			list_add_tail(&fmr->list, &pool->dirty_list);
-			++pool->dirty_len;
-			wake_up_process(pool->thread);
+			if (++pool->dirty_len >= pool->dirty_watermark) {
+				atomic_inc(&pool->req_ser);
+				wake_up_process(pool->thread);
+			}
 		}
 	}
 

-- 
Olaf Kirch  |  --- o --- Nous sommes du soleil we love when we play
okir at lst.de |    / | \   sol.dhoop.naytheet.ah kin.ir.samse.qurax