[ofa-general] [PATCH RFC v3 1/2] RDMA/Core: MEM_MGT_EXTENSIONS support

[Tom Tucker]

On Tue, 2008-05-27 at 10:33 -0500, Tom Tucker wrote:
> On Mon, 2008-05-26 at 16:02 -0700, Roland Dreier wrote:
> > > The "invalidate local stag" part of a read is just a local sink side
> >  > operation (ie no wire protocol change from a read).  It's not like
> >  > processing an ingress send-with-inv.  It is really functionally like a
> >  > read followed immediately by a fenced invalidate-local, but it doesn't
> >  > stall the pipe.  So the device has to remember the read is a "with inv
> >  > local stag" and invalidate the stag after the read response is placed
> >  > and before the WCE is reaped by the application.
> > 
> > Yes, understood.  My point was just that in IB, at least in theory, one
> > could just use an L_Key that doesn't have any remote permissions in the
> > scatter list of an RDMA read, while in iWARP, the STag used to place an
> > RDMA read response has to have remote write permission.  So RDMA read
> > with invalidate makes sense for iWARP, because it gives a race-free way
> > to allow an STag to be invalidated immediately after an RDMA read
> > response is placed, while in IB it's simpler just to never give remote
> > access at all.
> > 
> 
> So I think from an NFSRDMA coding perspective it's a wash...
> 
> When creating the local data sink, We need to check the transport type.
> 
> If it's IB --> only local access,
> if it's iWARP --> local + remote access.
> 
> When posting the WR, We check the fastreg capabilities bit + transport type bit:
> If fastreg is true -->
> 	Post FastReg
> 	If iWARP (or with a cap bit read-with-inv-flag)
> 		post rdma read w/ invalidate
> 	else /* IB */
> 		post rdma read

Steve pointed out a good optimization here. Instead of fencing the RDMA
READ here in advance of the INVALIDATE, we should post the INVALIDATE
when the READ WR completes. This will avoid stalling the SQ. Since IB
doesn't put the LKEY on the wire, there's no security issue to close. We
need to keep a bunch of fastreg MR around anyway for concurrent RPC.

Thoughts?
Tom

> 		post invalidate
> 	fi
> else
> 	... today's logic
> fi	
> 
> I make the observation, however, that the transport type is now overloaded
> with a set of required verbs. For iWARP's case, this means rdma-read-w-inv,
> plus rdma-send-w-inv, etc... This also means that new transport types will
> inherit one or the other set of verbs (IB or iWARP).
> 
> Tom
> 
> 
> >  - R.
> > _______________________________________________
> > general mailing list
> > general at lists.openfabrics.org
> > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
> > 
> > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
> 
> _______________________________________________
> general mailing list
> general at lists.openfabrics.org
> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
> 
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general