[dat-discussions] [openib-general] [RFC] DAT 2.0 immediatedataproposal
Davis, Arlin R
arlin.r.davis at intel.com
Mon Feb 6 13:16:45 PST 2006
I just want to get consensus on the requirements before we get too far.
One thing I forgot is that with Infiniband, the receive with immediate
provides the size of the rdma write that just completed. I think we
should include this in the requirements since there is ULP value here.
-arlin
>-----Original Message-----
>From: Kanevsky, Arkady [mailto:Arkady.Kanevsky at netapp.com]
>Sent: Monday, February 06, 2006 11:08 AM
>To: Kanevsky, Arkady; Davis, Arlin R; Sean Hefty
>Cc: dat-discussions at yahoogroups.com; openib-general at openib.org
>Subject: RE: [dat-discussions] [openib-general] [RFC] DAT 2.0
immediatedataproposal
>
>Arlin,
>It is too strong to state that Consumer should never send a message
>equal in size to the size of immediate data.
>Consumer knows from the context which one it is.
>it may be based on dedicated connection, or based on ULP protocol
>ordering.
>Arkady
>
>Arkady Kanevsky email: arkady at netapp.com
>Network Appliance Inc. phone: 781-768-5395
>1601 Trapelo Rd. - Suite 16. Fax: 781-895-1195
>Waltham, MA 02451 central phone: 781-768-5300
>
>
>> -----Original Message-----
>> From: Kanevsky, Arkady
>> Sent: Monday, February 06, 2006 2:05 PM
>> To: Davis, Arlin R; Sean Hefty
>> Cc: dat-discussions at yahoogroups.com; openib-general at openib.org
>> Subject: RE: [dat-discussions] [openib-general] [RFC] DAT 2.0
>> immediatedataproposal
>>
>> Arlin,
>> On Friday we agreed that receiver can not distinguish between
>> 4 byte of Send or 4 bytes of Immediate data if RDMA Write
>> with Immed is implemented as 2 operations:
>> RDMA Write followed by Send.
>>
>> ULP Reciever "expects" Immediate data that is why it posts
>> Recv. Depending on Transport capability it MAY complete as
>> Recv or as Recv_RDMA_Write_with_Immed_in_event.
>>
>> Neither Provider not Consumer can distinguish between the
>> cases unless there is additional info.
>>
>> Arkady
>>
>> Arkady Kanevsky email: arkady at netapp.com
>> Network Appliance Inc. phone: 781-768-5395
>> 1601 Trapelo Rd. - Suite 16. Fax: 781-895-1195
>> Waltham, MA 02451 central phone: 781-768-5300
>>
>>
>> > -----Original Message-----
>> > From: Davis, Arlin R [mailto:arlin.r.davis at intel.com]
>> > Sent: Monday, February 06, 2006 1:25 PM
>> > To: Kanevsky, Arkady; Sean Hefty
>> > Cc: dat-discussions at yahoogroups.com; openib-general at openib.org
>> > Subject: RE: [dat-discussions] [openib-general] [RFC] DAT 2.0
>> > immediate dataproposal
>> >
>> >
>> > Arkady,
>> >
>> > Your requirements are slightly different then the proposed set of
>> > requirements.
>> >
>> > "iii) DAPL Provider does not provide any identification
>> that that the
>> > Receive operation matches remote RDMA Write with Immediate
>> data if it
>> > completes as Receive DTO.
>> >
>> > - It is up to an ULP to separate Receive completion of remote
>> > Send from remote RDMA Write with Immediate Data."
>> >
>> > Tell me how this is possible? How can the application distinguish
>> > between a 4 byte message and a 4 byte immediate data
>> message? We would
>> > have to add a new requirement... "If the provider supports
>> immediate
>> > data in the payload the ULP cannot send a message equal to the
>> > immediate
>> > data size".
>> >
>> > -arlin
>> >
>> > >-----Original Message-----
>> > >From: Kanevsky, Arkady [mailto:Arkady.Kanevsky at netapp.com]
>> > >Sent: Monday, February 06, 2006 8:08 AM
>> > >To: Sean Hefty; Davis, Arlin R
>> > >Cc: dat-discussions at yahoogroups.com; openib-general at openib.org
>> > >Subject: RE: [dat-discussions] [openib-general] [RFC] DAT
>> > 2.0 immediate
>> > dataproposal
>> > >
>> > >Here are the changes to the existing requirements chapters
>> for RDMA
>> > >Write with Immediate Data.
>> > >
>> > >Feedback please.
>> > >Arkady
>> > >
>> > >Arkady Kanevsky email: arkady at netapp.com
>> > >Network Appliance Inc. phone: 781-768-5395
>> > >1601 Trapelo Rd. - Suite 16. Fax: 781-895-1195
>> > >Waltham, MA 02451 central phone: 781-768-5300
>> > >
>> > >
>> > >> -----Original Message-----
>> > >> From: Sean Hefty [mailto:mshefty at ichips.intel.com]
>> > >> Sent: Friday, February 03, 2006 7:30 PM
>> > >> To: Davis, Arlin R
>> > >> Cc: dat-discussions at yahoogroups.com; openib-general at openib.org
>> > >> Subject: Re: [dat-discussions] [openib-general] [RFC] DAT 2.0
>> > >> immediate dataproposal
>> > >>
>> > >> Davis, Arlin R wrote:
>> > >> > "Applications need an optimized mechanism to notify the
>> > >> receiving end
>> > >> > that RDMA write data has completed beyond the two
>> > operation method
>> > >> > currently used (RDMA write followed by message send).
>> > This new RDMA
>> > >> > write feature will support 4-bytes of inline data that
>> > will be sent
>> > >>
>> > >> Is there any reason to restrict the size of the immediate data?
>> > >> Could you define the API such that the size is variable?
>> I.e. the
>> > >> provider can simply give the immediate data size, with 0
>> > indicating
>> > >> that it is not supported.
>> > >>
>> > >> > It should avoid
>> > >> > any latency penalties normally associated with a two
>> > >> operation method.
>> > >>
>> > >> I would state this as a requirement. A write followed by a send
>> > >> should be pushed to the application, since they may be able to
>> > >> provide additional optimizations (such as combining
>> > >> operations) beyond what a provider could.
>> > >>
>> > >> > The initiating side must expose a 4-byte immediate data
>> > >> parameter for
>> > >> > the application to set the inline data. The receiving
>> side must
>> > >> > provide a mechanism to accept the 4-byte immediate
>> data. On the
>> > >> > receiving side, the write with immediate completion
>> > notification is
>> > >> > indicated through a receive completion. It is the
>> > responsibility of
>> > >> > the provider to identify to the application 4-byte
>> > >> immediate data from
>> > >> > a normal 4-byte send message. The inline byte ordering is
>> > >> application specific."
>> > >>
>> > >> Requirements look good to me.
>> > >>
>> > >> - Sean
>> > >> _______________________________________________
>> > >> openib-general mailing list
>> > >> openib-general at openib.org
>> > >> http://openib.org/mailman/listinfo/openib-general
>> > >>
>> > >> To unsubscribe, please visit
>> > >> http://openib.org/mailman/listinfo/openib-general
>> > >>
>> >
>> _______________________________________________
>> openib-general mailing list
>> openib-general at openib.org
>> http://openib.org/mailman/listinfo/openib-general
>>
>> To unsubscribe, please visit
>> http://openib.org/mailman/listinfo/openib-general
>>
More information about the general
mailing list