[ofa-general] iSer and Direct IO

[Cameron Harr]

Hi, I've been trying to compare performances between iSer and srpt and
am getting mixed results where iSer wins for IOPs and srpt wins for some
streaming b/w tests. I've tested with iozone, spew and FIO, and IOP
numbers are always higher on iSer. My problem though is that I'm a
little suspicious of some of the iSer numbers and whether they are
really using Direct IO. For example, you'll see below in some of my FIO
results that I'm getting a write B/W of 799.1 MB/s at one point. That's
way above what I can get natively on the device (~650 MB/s DIO) and is
more along the lines of buffered IO. If the IOP numbers are also using
some kind of caching, that could possibly taint them also. Does anyone
know if specifying DIO will really bypass all buffers or if something is
getting cached in the agents (iscsi, tgtd)?


FIO
--------------- iSer 1----iSer 2----SRPT 1----SRPT 2-
RBW (MB/s)      565.3     836.5     622.0     581.7
Read IOPs       63488.1   68053.8   5335.6    5446.1
WBW (MB/s)      799.1     737.7     589.5     594.4
Write IOPs      79086.6   80005.7   33884.6   34058.6


Thanks much,
Cameron