[dat-discussions] [openib-general] [RFC] DAT2.0immediatedataproposal

[Kanevsky, Arkady]

Caitlin,
can you clarify this.
Are you proposing that Consumer encode a bit of Immediate Data to
specify that it is immediate data?
iWARP will pass it in Send message and IB in Immediate Data.

Arkady Kanevsky                       email: arkady at netapp.com
Network Appliance Inc.               phone: 781-768-5395
1601 Trapelo Rd. - Suite 16.        Fax: 781-895-1195
Waltham, MA 02451                   central phone: 781-768-5300
 

> -----Original Message-----
> From: Caitlin Bestler [mailto:caitlinb at broadcom.com] 
> Sent: Thursday, February 09, 2006 2:40 PM
> To: Arlin Davis; Roland Dreier
> Cc: dat-discussions at yahoogroups.com; openib-general at openib.org
> Subject: RE: [dat-discussions] [openib-general] [RFC] 
> DAT2.0immediatedataproposal
> 
> openib-general-bounces at openib.org wrote:
> > Roland Dreier wrote:
> > 
> >> 
> >> Hmm.  Can you put a number on how much better RDMA write with 
> >> immediate is on current HCA hardware?  How does using the 
> underlying 
> >> OpenIB verbs ability to post a list of work requests compare (ie 
> >> posting an RDMA write followed by a send in one verbs call)?
> >> Maybe "post multiple" is a better direction for DAT.
> >> 
> >> 
> > With post multiple, unlike immediate data, you don't have 
> the ability 
> > to distinguish between a normal receive and a rdma write completion 
> > indication on the other end. This is the uniqueness of the service 
> > that cannot be provided by the post multiple. Yes, post 
> multiple would 
> > be a nice option for DAT it is just a different service. It 
> would also 
> > be required to conform to the semantics rules of the bundled 
> > operations so you could not do any optimization tricks under the 
> > covers with an IB rdma_write_immediate operation.
> > 
> 
> A post_multiple also requires defining a single "DTO" data 
> structure. If the post multiple is atomic (meaning all make 
> it or none do) then it requires an intermediate data 
> structure to have been created. If it is not atomic there 
> really isn't reason for it to not just be a utility function 
> layered above DAT.
> 
> What I'm not seeing with the immediate is this urgent need by 
> the application to be able to use the same 32-bit value for 
> both an immediate and a 4 byte message that requires an 
> entire additional API just to support it.  Why can't the 
> application just add a bool to the send message?
> Or encode the 32-bits so that they come from disjoint domains?
> 
> There seems to be agreement that a consolidated 
> write-and-send call would enable the application to get the 
> benefits of rdma write with immediate whenever the 
> application could distinguish the two.
> 
> I cannot see why doing this is almost free for virtually all 
> applications, and trivial for the remainder. Adding and 
> documenting an extra call to deal with such an extreme corner 
> case that is being presented only in the abstract is just not 
> justified. This extra capability has to have enough 
> functionality for enough applications to justify keeping it 
> on the books, writing test cases for it, etc.
> 
> We already made a similar decision in having a 128-bit IA 
> Address. That means we cannot support a host that interfaces 
> to the Internet with IPv6 and an InfiniBand network that not 
> only had global GIDs, but allocated a global subnetwork a 
> network id that was already in use as a valid public IPv6 network.
> 
> The complexity of dealing with an IA Address that was
> 128+1 bits was simply not jusitified to deal with
> an extreme corner case that could very easily be avoided 
> (there is no shortage of "site local" network IDs in the 
> IPv6/GID format, so using a global network prefix that was 
> disjoint from the official IPv6 hierarchy would be just plain silly).
> 
> So far I haven't seen any explanation as to why an 
> application has a need to encode this 33rd bit of their 
> message in this terribly transport specific matter. Is there 
> some severe performance penalty to slightly restructuring the 
> send message so that it is no longer ambiguous with the 
> immeidate data?
> 
> 
> 
>  
> Yahoo! Groups Links
> 
> <*> To visit your group on the web, go to:
>     http://groups.yahoo.com/group/dat-discussions/
> 
> <*> To unsubscribe from this group, send an email to:
>     dat-discussions-unsubscribe at yahoogroups.com
> 
> <*> Your use of Yahoo! Groups is subject to:
>     http://docs.yahoo.com/info/terms/
>  
> 
>