[dat-discussions] RE: [openib-general] RE: [RFC] DAT 2.0 immediate data proposal

[Kanevsky, Arkady]

Caitlin,
Agree that Send with immed is too hard to handle.
I have not heard from any ULP that they need that.
So we can take informal vote and close that issue.

The sizing of EVD to handle 2 completions in case of the error
for post of RDMA_write_with_immed can be handled by Provider
adding extra if EVD will be used for posting RDMA_write_with_immed.
It does not allow Consumer to "optimize" queue size based on "exact"
number of oustanding RDMA_write_with_immed ops but it is simpler
to program to.

Of course ULP can be "adaptive" and chooses the code pass
based on Provider attr if we add the attr if extra queue size is needed.
It is separate from how immed data is returned.
We can combined the 2 under one Provider attr but conceptually it is
wrong
to combine two.


Arkady Kanevsky                       email: arkady at netapp.com
Network Appliance Inc.               phone: 781-768-5395
1601 Trapelo Rd. - Suite 16.        Fax: 781-895-1195
Waltham, MA 02451                   central phone: 781-768-5300
 

> -----Original Message-----
> From: Caitlin Bestler [mailto:caitlinb at broadcom.com] 
> Sent: Friday, January 27, 2006 12:23 PM
> To: dat-discussions at yahoogroups.com; Kanevsky, Arkady; Arlin Davis
> Cc: Lentini, James; openib-general at openib.org
> Subject: RE: [dat-discussions] RE: [openib-general] RE: [RFC] 
> DAT 2.0 immediate data proposal
> 
> dat-discussions at yahoogroups.com wrote:
> >> But this penalizes user which need to deal with 2 way to deal with 
> >> post calls and completions.
> >> 
> >> I do not think we are not to far from consensus.
> >> Transport independent App will allocate 4 bytes extra for buffers 
> >> that can match immediate data. Completion data will return 
> where the 
> >> immediate data is return (Consumer can not request it on posting), 
> >> and 4 bytes for immediate data in completion event. The rest are 
> >> ironing details for complete specification.
> >> This is no different than for any other new functionality proposed.
> >> And except for wasting 4 bytes per buffer or completion I 
> do not see 
> >> how it penalizes IB. Moreover if Apps knows that Provider returns 
> >> immediate data in completion event it can avoid any penalty.
> > 
> > There is no penalty to the user if you just provide native features 
> > via extensions. Your extension will provide the best possible 
> > interface for your native capabilities.
> > 
> > I think we are further from consensus then we first thought:
> > 
> > Right now we have a new post recv, different delivery 
> mechanisms, and 
> > a requirement to allocate an extra 4 bytes of user data.
> > 
> > The only requirement to support immediate data on IB, is a new post 
> > send and write immediate data calls and a new event data construct. 
> > The normal post_recv can be used unchanged and can already process 
> > normal and immediate data. No requirement on the user to 
> allocate and 
> > manage an extra 4 bytes in the receive buffer. In fact, you 
> can post 
> > receive with no buffer.
> > 
> > In order to support immediate data via iWARP, you now have a 
> > requirement to use a special new receive post, new user buffer 
> > constructs to place the data, and new delivery method that 
> has to be 
> > checked via provider attributes or at event time.
> > 
> > Is there anyway to get this closer? If not, I would recommend going 
> > back to an extension interface for immediate data.
> > 
> 
> I think the trick to finding out if there is something useful 
> that can be made transport neutral is to work in the opposite 
> direction.
> 
> Start with the message sequence that the application would use
> *without* immediates, and then ask if there is a way to allow 
> an InfiniBand Provider to compress that message sequence.
> 
> That is possible for RDMA Write with Immediate. With careful 
> definition of a composite message it can be viewed as a 
> transport specific replacement for an RDMA Write followed by 
> a 4-byte RDMA Send. There are only two special considerations 
> required:
> 
> 1) A single post has to submit the combination (otherwise it 
>    is too difficult for the Provider to detect the optimization).
> 2) The receive completion may report the received data in the
>    user supplied buffer OR in an "immediate data" field in the
>    completion.
> 
> I do not think it is feasible to define a transport neutral 
> equivalent of a RDMA Send with Immediate. How is the extra 
> data transmitted via iWARP? An extra send? Pre-pend the four 
> bytes? Or 4 bytes at the end? Delivery of the immediate data 
> is transport dependent?
> 
> Adding an immediate data field to the completion doesn't cost 
> much, and it would allow IB DAT Provider to interact with 
> IB-specific fields. But I can't see adding a send with 
> immediate method in any way that would create an expectation 
> in developers that it would work in a transport neutral fashion.
> 
> Write with immediate is possible. It carries the complexity 
> that a single DTO request might result in two flushed work 
> completions.
> The current consensus is that this was too complex relative 
> to the benefit. But that's really a call for application developers.
> It isn't that hard for an iWARP Provider to implement. The 
> question is whether given the complexity of properly sizing 
> the QP that it will actually be used.
>