[ofa-general] mapping kernel memory to userspace in <= 2.6.14

[Steve Wise]

On Tue, 2007-03-13 at 09:08 -0500, Steve Wise wrote:
> Hey Roland,
> 
> Remember my little ofed 1.2 bug where the chelsio WQ and CQ memory
> aren't getting mapped to userspace correctly for RHEL4U4?  Well through
> experimentation I've shown that it fails all the way through 2.6.14 and
> works fine with 2.6.15 and beyond.  
> 
> Perusing the mm/memory.c log from v2.6.14..v2.6.15 shows lots of
> changes.  Many of the comments talk about no longer needing to set the
> reserved bit on page entries.  On a hunch I hacked in calling
> SetPageReserved() on each page entry for the memory allocated via
> dma_alloc_coherent().  And BOOM...things start working.  Below is a
> patch to show the hack.
> 
> Does this make sense to you?  I'm not a VM expert.  
> 
> Also, looking at other 2.6.14 drivers, I see that many of them seem to
> do this same trick apparently for making sure the pages aren't swapped.
> However, they also clear the bit before freeing the memory, which makes
> sense.  My original hack had a ClearPageReserved() loop before freeing
> my dma coherent memory, but I got crashes on process exit where the map
> count on pages wasn't correct (the refcnt went to -1 apparently). It hit
> the BUG_ON() in page_remove_rmap()  at line 487 in mm/rmap.c (2.6.14.7
> kernel).  
> 
> So I'm thinking this hack isn't quite correct.  Got and ideas?
> 

I figured out why I was hitting the BUG_ON() in page_remove_rmap().  Its
because my library was destroying the QP or CQ object _before_ unmaping
the objects.  I changed it to unmap first, and things work as expected.

So my conclusion is that SetPageReserved() is needed to map kernel
memory into userspace for kernels older than 2.6.15.  

My guess is ehca and ipath have similar issues on these older kernels. 


Steve.