[Openib-windows] Windows DMA model

[Fab Tillier]

> From: Leonid Keller [mailto:leonid at mellanox.co.il]
> Sent: Tuesday, January 17, 2006 3:58 AM
> 
> First, to inform: I've found the problem with my kernel DMA tesing.
> It seems like a bug in Microsoft: when you ask map registers for a 2GB
> transfer - which is the max for our cards - IoGetDmaAdapter returns 1
> register. For *any* number X, less than that, they return an appropriate
> number.

Did you check to see what happens if you make a request larger than 2GB?  Is 2GB
the threshold?

In any case, I'm glad to hear this is working now.

> It seems like gives us opportunity to solve all the problems for kernel
> work.

Yes, it should.  However, kernel clients will need to do the DMA mappings before
and unmap after each I/O so that things work properly.  Clients doing only local
operations (not exposing buffers for RDMA) can create a DMA MR (for all of
physical memory), and then only need to perform the DMA mappings during I/O.

Any ring buffers shared between the HCA driver and hardware will need to be
allocated via AllocateCommonBuffer.

> As for the userland, I also don't see a solution, but giving up
> os-bypassing and performing memory registration during send/recv.

For user-land, I'm thinking more and more that having the kernel allocate common
buffers for the ring buffers (in CQs and QPs) is the way to go.  I don't know
how well this will scale, but it will guarantee that the ring buffers at least
are working properly.

For I/O, it would all depend on whether the platform performs double buffering
or not.  However, there's no way to tell what the platform is doing under the
DMA APIs, so you're right - for guaranteed correct operation, the app would have
to register/deregister for each send/recv.  However, this goes against the
design of WSD, so I think they have some explaining to do.  I'll follow up with
them and see if I can get any additional information.

- Fab