[openib-general] [PATCHv2] mthca: speed up memory registration by filling MTTs directly

[Roland Dreier]

I was going to apply this, but then I realized that mthca is screwed
up on non-cache-coherent CPUs with memfree HCAs, and this patch makes
things much worse.  The problem is that we allocate the MTT table with
alloc_pages() and then do pci_map_sg().  But there's no
pci_dma_sync_sg calls when the CPU tries to write directly to the MTT
table, and in fact not even that would work: since a
non-cache-coherent CPU can only work on cacheline-sized chunks there's
no safe way to touch the MTT table.

What all that means is that FMRs are currently broken for memfree on
non-coherent CPUs.  And this patch would break all memory
registration.  I think the fix has to be to use dma_alloc_coherent()
to allocate the pages for the MTT table (and any other table allocated
in lowmem -- but I don't think there are any others).

Unfortunately my PowerPC 440 system is being reworked right now so I
can't test this for a few days.

I think this still can go into 2.6.20 after -rc1 if we can get this
fixed up.

 - R.