[dat-discussions] [openib-general] [RFC]DAT2.0immediatedataproposal

[Caitlin Bestler]

-----Original Message-----
From: openib-general-bounces at openib.org on behalf of Sean Hefty
Sent: Wed 2/8/2006 11:20 PM
To: 'Roland Dreier'; Arlin Davis
Cc: dat-discussions at yahoogroups.com; openib-general at openib.org
Subject: RE: [dat-discussions] [openib-general] [RFC]DAT2.0immediatedataproposal
 
>Hmm.  Can you put a number on how much better RDMA write with
>immediate is on current HCA hardware?  How does using the underlying
>OpenIB verbs ability to post a list of work requests compare (ie
>posting an RDMA write followed by a send in one verbs call)?
>Maybe "post multiple" is a better direction for DAT.

A "post multiple" call as a general API makes sense, but I think that's a
separate issue.

Given that IB provides true immediate data with RDMA writes, a way should be
available to make use of it.  I don't know what the performance numbers between
using a write with immediate versus a write followed by a send, but I don't
think that anyone could argue that the write with immediate wouldn't perform
better.

To me, the question is whether write with immediate is supported as a transport
specific extension, which was Arlin's original patch, or through some standard
API.  The attempt to make the API standard, so that iWarp could emulate it
(poorly in my view), is what appears to be driving the disagreements.

It also appears to me that the decisions are coming down to one of the
following.  If iWarp can emulate write with immediate, then a generic API should
be used.  If iWarp cannot properly emulate write with immediate, then the API
should be transport specific.  It's curious to me that in both cases, iWarp is
driving the API decision and design for something that is an IB specific
feature.

- Sean

----------------------

But why define an IB specific feature when a transport neutral feature
can be defined?

Viewing the operation as Write with following Send maintains transport
neutral semantics AND allows IB to encode it as a Write with Immediate.

That avoids IB to use the silicon that already exists to support compressing
the Write and Send into a single message. That is the real benefit, isn't it?
And for both transports it enables the Provider to pass the 4 byte immediate
data by value rather than by registered reference. So there is a definite benefit
to IB, and a potential benefit to IP, and it works for both transports.

The *only* thing gained by making it a transport specific method is
the implicit 33rd bit in the "that RDMA Write payload you asked for
has arrived" message.

Is there a concrete example of any benefit from encoding a 33rd bit in
the selection of Write with Immediate versus Write followed by 32-bit Send?