<html>
<body>
<font size=3>At 05:10 PM 3/20/2006, Fabian Tillier wrote:<br>
<blockquote type=cite class=cite cite="">On 3/20/06, Talpey, Thomas
<Thomas.Talpey@netapp.com> wrote:<br>
> Ok, this is a longer answer.<br>
><br>
> At 06:08 PM 3/20/2006, Fabian Tillier wrote:<br>
> >As to using FMRs to create virtually contiguous regions, the
last data<br>
> >I saw about this related to SRP (not on OpenIB), and resulted in
a<br>
> >gain of ~25% in throughput when using FMRs vs the "full
frontal" DMA<br>
> >MR. So there is definitely something to be gained by
creating<br>
> >virutally contiguous regions, especially if you're doing a lot
of RDMA<br>
> >reads for which there's a fairly low limit to how many can be
in<br>
> >flight (4 comes to mind).<br>
><br>
> 25% throughput over what workload? And I assume, this was with
the<br>
> "lazy deregistration" method implemented with the current
fmr pool?<br>
> What was your analysis of the reason for the improvement - if it
was<br>
> merely reducing the op count on the wire, I think your issue lies
elsewhere.<br><br>
This was a large block "read" workload (since HDDs typically
give<br>
better read performance than write). It was with lazy
deregistration,<br>
and the analysis was that the reduction of the op count on the wire<br>
was the reason. It may well have to do with how the target chose
to<br>
respond, though, and I have no idea how that side of things was<br>
implemented. It could well be that performance could be
improved<br>
without going with FMRs.</font></blockquote><br>
Quite often performance is governed by the target more than the initiator
as it is in turn governed by its local cache and disc mech performance /
capacity. Large data movements typically are a low op count from
the initiator perspective therefore it seems a bit odd to state that
performance can be dramatically impacted by the op count on the
wire.<br><br>
<br>
<blockquote type=cite class=cite cite=""><font size=3>> Also, see
previous paragraph - if your SRP is fast but not safe, then only<br>
> fast but not safe applications will want to use it. Fibre channel
adapters<br>
> do not introduce this vulnerability, but they go fast. I can show
you NFS<br>
> running this fast too, by the way.<br><br>
Why can't Fibre Channel adapters, or any locally attached hardware
for<br>
that matter, DMA anywhere in memory? Unless the chipset
somehow<br>
protect against it, doesn't locally attached hardware have free
reign<br>
over DMA?</font></blockquote><br>
As a general practice, future volume I/O chipsets across multiple market
segments will implement an IOMMU to restrict where DMA is allowed.
Both AMD and Intel have recently announced specifications to this effect
which reflect what has been implemented in many non-x86 chipset
offerings. Whether a given OS always requires this protection to be
enabled is implementation-specific but it is something that many within
the industry and customer base require.<br><br>
Mike<br><br>
<br>
<blockquote type=cite class=cite cite=""><font size=3>Also, please don't
take my anectdotal benchmark results as an<br>
endorsement of the Mellanox FMR design - the data was presented to
me<br>
by Mellanox as a reason to add FMR support to the Windows stack
(which<br>
currently uses the "full frontal" approach due to limitations
of the<br>
verbs API and how it needs to be used for storage). I never had
a<br>
chance to look into why the gains where so large, and it could be<br>
either the SRP target implementation, a hardware limitation, or a<br>
number of other issues, especially since a read workload results in<br>
RDMA Writes from the target to the host which can be pipelined much<br>
deeper than RDMA Reads.<br><br>
- Fab<br>
_______________________________________________<br>
openib-general mailing list<br>
openib-general@openib.org<br>
<a href="http://openib.org/mailman/listinfo/openib-general" eudora="autourl">
http://openib.org/mailman/listinfo/openib-general</a><br><br>
To unsubscribe, please visit
<a href="http://openib.org/mailman/listinfo/openib-general" eudora="autourl">
http://openib.org/mailman/listinfo/openib-general</a>
</font></blockquote></body>
</html>