[openib-general] Re: Re: [PATCH] (repost) sdp: replace mlock with get_user_pages

[Libor Michalek]

On Wed, May 25, 2005 at 11:21:28PM +0300, Michael S. Tsirkin wrote:
> Quoting r. Libor Michalek <libor at topspin.com>:
> > Subject: Re: Re: [PATCH] (repost) sdp: replace mlock with get_user_pages
> > 
> > On Fri, May 13, 2005 at 04:51:45PM +0300, Michael S. Tsirkin wrote:
> > > Quoting r. Roland Dreier <roland at topspin.com>:
> > > > 
> > > >     Libor>   Always call aio_complete() immediately when
> > > >     Libor> iocb_complete() is called, and only spawn the work thread
> > > >     Libor> to unlock the memory after the aio_complete() call. The
> > > >     Libor> patch is below.
> > > 
> > > Libor, I dont think its a good idea - this will break other assumptions,
> > > like the assumption that the task mm isnt destroyed before we unlock
> > > the memory.
> > 
> >   That's a good point.
> > 
> > > > Another alternative would be to always complete aios asynchronously,
> > > > which should preserve the order.  I guess this would hurt latency for
> > > > small ios...
> > > 
> > > To avoid hurting latency, lets count the number of outstanding
> > > asynchronous AIOs, and if there are asynchronous AIOs complete
> > > all of them asynchronously.
> > > 
> > > Does this make sense?
> > 
> >   Yes, except that the current iocb code does not reference individual
> > sockets anywhere, and do_iocb_complete would have to be the function
> > which decremented the per connection counter of outstanding AIOs. Also,
> > since we don't have to do get_user_pages a second time on the send
> > path, this would only need to be done on the recv path.
> 
> I thought about this some more: what if we set users to 1 before
> releasing the irq spinlock, and call sdp_conn_unlock in thread context
> after completing the aio iocb?
> Any synchronous transfer would then wait till socket is unloacked.

  So use 'users' as a reference count, and basically increment the
lock before spawning do_iocb_complete. do_iocb_complete would then
unlock one reference and the function calling iocb_complete would 
unlock the other reference? This could work... Remeber that it's
possible for many do_iocb_complete functions to be in flight for a
given connection. 

  FYI, The problem I saw occured when the iocb_complete was called 
during the sdp_conn_unlock CQ poll, which is done with IRQ diabled.

-Libor