[nvmewin] IO queue memory

[Luse, Paul E]

So apparently with this API non-cached does not imply non-paged which I suppose makes sense as the cache mode here refers to CPU cache.  I don't see a way to get non-paged NUMA node specific contiguous memory at all actually, anyone?

From: Luse, Paul E
Sent: Wednesday, July 18, 2012 5:44 AM
To: nvmewin at lists.openfabrics.org
Subject: IO queue memory

Discussion point I wanted to get some input on:

Memory type:  When we designed this, we chose cached memory for our IO queues because we don't have to worry about DMA coherency with IA anymore however the implication here is that our queues can now be paged out which I don't think we want for performance reasons.  Also, if we don't decide to switch to non-paged for that reason we need to rework (minor) our shutdown code which is touching IO queue memory at DIRQL which, of course, you can't do.  I think for the paging reason alone we should consider non cached allocations for the IO queues.  Other thoughts?

We may want to also think about a different strategy for IO queue sizing as well, if we switch to non cached, to be a little more accurate/conservative with how much memory we're using based on the current config.  Right now, for example, on a 32 core system we'll use 2MB of memory just for IO queues.

Thx
Paul
____________________________________
Paul Luse
Sr. Staff Engineer
PCG Server Software Engineering
Desk: 480.554.3688, Mobile: 480.334.4630

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/nvmewin/attachments/20120718/cb3ad82d/attachment.html>