[dat-discussions] RE: [openib-general] RE: [RFC] DAT 2.0 immediate data proposal

[Caitlin Bestler]

dat-discussions at yahoogroups.com wrote:
>> But this penalizes user which need to deal with 2 way to deal with
>> post calls and completions. 
>> 
>> I do not think we are not to far from consensus.
>> Transport independent App will allocate 4 bytes extra for buffers
>> that can match immediate data. Completion data will return where the
>> immediate data is return (Consumer can not request it on posting),
>> and 4 bytes for immediate data in completion event. The rest are
>> ironing details for complete specification. 
>> This is no different than for any other new functionality proposed.
>> And except for wasting 4 bytes per buffer or completion I do not see
>> how it penalizes IB. Moreover if Apps knows that Provider returns
>> immediate data in completion event it can avoid any penalty.
> 
> There is no penalty to the user if you just provide native
> features via extensions. Your extension
> will provide the best possible interface for your native capabilities.
> 
> I think we are further from consensus then we first thought:
> 
> Right now we have a new post recv, different delivery
> mechanisms, and a requirement to allocate an extra 4 bytes of user
> data. 
> 
> The only requirement to support immediate data on IB, is a
> new post send and write immediate data calls and a new event
> data construct. The normal post_recv can be used unchanged
> and can already process normal and immediate data. No
> requirement on the user to allocate and manage an extra 4
> bytes in the receive buffer. In fact, you can post receive
> with no buffer.
> 
> In order to support immediate data via iWARP, you now have a
> requirement to use a special new receive post, new user
> buffer constructs to place the data, and new delivery method
> that has to be checked via provider attributes or at event time.
> 
> Is there anyway to get this closer? If not, I would recommend
> going back to an extension interface for immediate data.
> 

I think the trick to finding out if there is something useful
that can be made transport neutral is to work in the opposite
direction.

Start with the message sequence that the application would use
*without* immediates, and then ask if there is a way to allow
an InfiniBand Provider to compress that message sequence.

That is possible for RDMA Write with Immediate. With careful
definition of a composite message it can be viewed as a transport
specific replacement for an RDMA Write followed by a 4-byte
RDMA Send. There are only two special considerations required:

1) A single post has to submit the combination (otherwise it 
   is too difficult for the Provider to detect the optimization).
2) The receive completion may report the received data in the
   user supplied buffer OR in an "immediate data" field in the
   completion.

I do not think it is feasible to define a transport neutral
equivalent of a RDMA Send with Immediate. How is the extra
data transmitted via iWARP? An extra send? Pre-pend the
four bytes? Or 4 bytes at the end? Delivery of the immediate
data is transport dependent?

Adding an immediate data field to the completion doesn't cost
much, and it would allow IB DAT Provider to interact with
IB-specific fields. But I can't see adding a send with immediate
method in any way that would create an expectation in developers
that it would work in a transport neutral fashion.

Write with immediate is possible. It carries the complexity that
a single DTO request might result in two flushed work completions.
The current consensus is that this was too complex relative to 
the benefit. But that's really a call for application developers.
It isn't that hard for an iWARP Provider to implement. The 
question is whether given the complexity of properly sizing
the QP that it will actually be used.