<html>
<body>
<font size=3>At 05:35 PM 3/14/2005, Caitlin Bestler wrote:<br>
<blockquote type=cite class=cite cite=""> <br><br>
> -----Original Message-----<br>
> From: Troy Benjegerdes
[<a href="mailto:hozer@hozed.org" eudora="autourl">
mailto:hozer@hozed.org</a>] <br>
> Sent: Monday, March 14, 2005 5:06 PM<br>
> To: Caitlin Bestler<br>
> Cc: openib-general@openib.org<br>
> Subject: Re: [openib-general] Getting rid of pinned memory
requirement<br>
> <br>
> > <br>
> > The key is that the entire operation either has to be fast
<br>
> > enough so that no connection or application session layer<br>
> > time-outs occur, or an end-to-end agreement to suspend the<br>
> > connetion is a requirement. The first option seems more<br>
> > plausible to me, the second essentially <br>
> > reuqires extending the CM protocol. That's a tall order even
for <br>
> > InfiniBand, and it's even worse for iWARP where the CM <br>
> > functionality typically ends when the connection is
established.<br>
> <br>
> I'll buy the good network design argument.</font></blockquote><br>
I and others designed InfiniBand RNR (Receiver not ready) operations to
allow one to adjust V-to-P mappings (not change the address that was
advertised) in order to allow an OS to safely play some games with memory
and not drop a connection. The time values associated with RNR
allow a solution to tolerate up to infinite amount of time to perform
such operations but the envisioned goal was to do this on the order of a
handful or milliseconds in the worse case. For iWARP, there was no
support for defining RNR functionality as indeed many people claimed one
could just drop in-bound segments and allow the retransmission protocol
to deal with the delay (even if this has performance implications due to
back-off algorithms though some claim SACK would minimize this to a large
extent). Again, the idea was to minimize the worse case to
milliseconds of down time. BTW, all of this assumed that the OS
would not perform these types of changes that often so the long-term
impact on an application would be minimum.<br><br>
<blockquote type=cite class=cite cite=""><font size=3>> <br>
> I suppose if the kernel wants to revoke a card's pinned <br>
> memory, we should be able to guarantee that it gets new <br>
> pinned memory within a bounded time. What sort of timing do <br>
> we need? Milliseconds?<br>
> Microseconds?<br>
><br>
> In the case of iWarp, isn't this just TCP underneath? If so, <br>
> can't we just drop any packets in the pipe on the floor and <br>
> let them get retransmitted? (I suppose the same argument goes <br>
> for infiniband..<br>
> what sort of a time window do we have for retransmission?)<br>
> <br>
> What are the limits on end-to-end flow control in IB and iWarp?<br>
> <br><br>
>From the RDMA Provider's perspective, the short answer is "quick
enough so that I don't have to do anything heroic to keep the connection
alive."</font></blockquote><br>
It should not require anything heroic. What is does require is a
local method to suspend the local QP(s) so that it cannot place or read
memory in the effected area. That can take some time depending upon
the implementation. There is then the time to over write the
mappings which again depending upon the implementation and the number of
mappings could be milliseconds in length.<br><br>
<blockquote type=cite class=cite cite=""><font size=3>With TCP you also
have to add "and healthy". If you've ever had a long download
that got effectively stalled by a burst of noise and you just hit the
'reload' button on your browser then you know what I'm talking
about.<br><br>
But in transport neutral terms I would think that one RTT is definitely
safe -- that much data could have<br>
been dropped by one switch failure or one nasty spike in inbound
noise.<br><br>
> > <br>
> > Yes, there are limits on how much memory you can mlock, or even
<br>
> > allocate. Applications are required to reqister memory
precisely <br>
> > because the required guarantess are not there by default. <br>
> Eliminating <br>
> > those guarantees *is* effectively rewriting every RDMA
application <br>
> > without even letting them know.<br>
> <br>
> Some of this argument is a policy issue, which I would argue <br>
> shouldn't be hard-coded in the code or in the network hardware.<br>
> <br>
> At least in my view, the guarantees are only there to make <br>
> applications go fast. We are getting low latency and high <br>
> performance with infiniband by making memory registration go <br>
> really really slow. If, to make big HPC simulation <br>
> applications work, we wind up doing memcpy() to put the data <br>
> into a registered buffer because we can't register half of <br>
> physical memory, the application isn't going very fast.<br>
><br><br>
What you are looking for is a distinction between registering<br>
memory to *enable* the RNIC to optimize local access and <br>
registering memory to enable its being advertised to the<br>
remote end.<br><br>
Early implementations of RDMA, both IB and iWARP, have not<br>
distinquished between the two. But theoretically *applications*<br>
do not need memory regions that are not enabled for remote<br>
access to be pinned. That is an RNIC requirement that could<br>
evolve. But applications themselves *do* need remotely<br>
accessible memory regions, portions of which they intend<br>
to advertise with RKeys, to be truly available (i.e., pinned).<br><br>
You are also making a policy assumption that an application<br>
that actually needs half of physical memory should be using<br>
paged memory. Memory is cheap, and if performance is critical<br>
why should this memory be swapped out to disk?<br><br>
Is the limitation on not being able to register half of<br>
physical memory based upon some assumption that swapping<br>
is a requirement? Or is it a limitation in the memory region<br>
size? If it's the latter, you need to get the OS to support<br>
larger page sizes.</font></blockquote><br>
For some OS, you can pin very large areas. I've seen 15/16 of
memory being able to be pinned with no adverse impacts on the
applications. For these OS, kernel memory is effectively pinned
memory. As such, depending upon the mix of services being provided,
the system may operate quite nicely with such large amounts of memory
being pinned. As more services are "ported" to operate
over RDMA technologies, memory management isn't necessarily any harder;
it just becomes something people have to think more about. Today's
VM designs have allowed people to get sloppy as they assume that swapping
will occur and since many platforms are not that loaded, they don't see
any real adverse impacts. User-space RDMA applications requires
people to think once again about memory management and that swapping
isn't a get-out-of-jail card. One needs to develop resource
management tools to determine who obtains specified amounts of resources
and their priorities. For the most part, this is somewhat a
re-invention of some thinking that went into the micro-kernel work in
past years. These problems are not intractable; they are only
constrained by the legacy inertia inherent in all technologies
today.<br><br>
<font size=3>Mike</font></body>
</html>