<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML xmlns="http://www.w3.org/TR/REC-html40" xmlns:v =
"urn:schemas-microsoft-com:vml" xmlns:o =
"urn:schemas-microsoft-com:office:office" xmlns:w =
"urn:schemas-microsoft-com:office:word" xmlns:st1 =
"urn:schemas-microsoft-com:office:smarttags"><HEAD>
<META http-equiv=Content-Type content="text/html; charset=us-ascii">
<META content="MSHTML 6.00.2900.2769" name=GENERATOR><o:SmartTagType
downloadurl="http://www.5iantlavalamp.com/" name="place"
namespaceuri="urn:schemas-microsoft-com:office:smarttags"></o:SmartTagType><!--[if !mso]>
<STYLE>st1\:* {
BEHAVIOR: url(#default#ieooui)
}
</STYLE>
<![endif]-->
<STYLE>@font-face {
font-family: Courier;
}
@page Section1 {size: 8.5in 11.0in; margin: 1.0in 1.25in 1.0in 1.25in; }
P.MsoNormal {
FONT-SIZE: 11pt; MARGIN: 3pt 0in; FONT-FAMILY: "Times New Roman"
}
LI.MsoNormal {
FONT-SIZE: 11pt; MARGIN: 3pt 0in; FONT-FAMILY: "Times New Roman"
}
DIV.MsoNormal {
FONT-SIZE: 11pt; MARGIN: 3pt 0in; FONT-FAMILY: "Times New Roman"
}
A:link {
COLOR: blue; TEXT-DECORATION: underline
}
SPAN.MsoHyperlink {
COLOR: blue; TEXT-DECORATION: underline
}
A:visited {
COLOR: purple; TEXT-DECORATION: underline
}
SPAN.MsoHyperlinkFollowed {
COLOR: purple; TEXT-DECORATION: underline
}
P.MsoPlainText {
FONT-SIZE: 10pt; MARGIN: 0in 0in 0pt; FONT-FAMILY: "Courier New"
}
LI.MsoPlainText {
FONT-SIZE: 10pt; MARGIN: 0in 0in 0pt; FONT-FAMILY: "Courier New"
}
DIV.MsoPlainText {
FONT-SIZE: 10pt; MARGIN: 0in 0in 0pt; FONT-FAMILY: "Courier New"
}
PRE {
FONT-SIZE: 10pt; MARGIN: 0in 0in 0pt; FONT-FAMILY: "Courier New"
}
SPAN.EmailStyle17 {
COLOR: windowtext; FONT-FAMILY: Arial; mso-style-type: personal-compose
}
DIV.Section1 {
page: Section1
}
</STYLE>
<!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></HEAD>
<BODY lang=EN-US vLink=purple link=blue>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>Arlin,</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>nice proposal, thanks.</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>I have one high level question and a few specific
technical ones.</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005></SPAN></FONT> </DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>1. Why do you want to provide this functionality via
extension instead of part of new DAT spec, say 2.0?</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>This will allow Consumers to use all events,
operations, and Provider/IA functionality uniformly instead</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>of via 2 separate layers. This will also ensure that
this basic funcionality can be provided by all DAPL Provider</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>the same way on DAPL and DAT
layers.</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>DAPL 2.0 is not done yet so we have time to incorporate
that.</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>DAPL 2.0 already introduced new functionality which is
easy to beef up for your proposal.</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT size=2><SPAN class=554400019-23122005><FONT
face=Arial><FONT color=#0000ff>See <FONT size=2>DAT_DTOS for example. DAT_EVENT
is also modified to handle remote invalidation
so</FONT></FONT></FONT></SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>a small addition for Immediate data and Atoimc ops is a
sensible addition.</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>This should simplify proposal significantly. As you
will not need to introduce any new</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>EXT structures.</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005></SPAN></FONT> </DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>In general, extension route was intended for RNIC|HCA
providers to expose HW capabilities beyond</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>IBTA, iWARP and VIA standards. The standard RDMA
functionality is best handle via spec addition.</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>DAT 2.0 does it for FMR, remote and local memory
invalidation as well as others.</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005></SPAN></FONT> </DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>I had posted a complete list of changes/addition to DAT
2.0 about a month ago.</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>But we had not discussed yet version change from 1.3 to
2.0 nor how much backwards compatibility spec</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>will provide.</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005></SPAN></FONT> </DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>2. What is IMMED_EVENT? is it just immediate data
without any payload one?</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>I suggest chnaging the name so it will not use "EVENT".
Just call it NO_PAYLOAD.</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>Do you want to support 2 different way to delivery
immediate data?</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>One in event and one in data
payload?</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>Why? I would think that just an event way will
do.</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005></SPAN></FONT> </DIV>
<DIV dir=ltr align=left><FONT size=2><SPAN class=554400019-23122005><FONT
color=#0000ff><FONT face=Arial>3. I suggest beefing up <FONT
size=2>DAT_DTO_COMPLETION_EVENT_DATA and DAT_DTOS
</FONT></FONT></FONT></SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>to convey which operation completed and return
Immediate data if complete operation had immediate data.</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>Since we already modified these 2 struct as part of DAT
2.0 change lets add your proposal to the change.</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>This will allow Consumers to use single approach to
deal with completions, extension to the current one</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>but not a structural one. No need for
DAT_EXTENSION_DATA, DAT_EXT_EVENT_TYPE, DAT_EXT_OP</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>nor the whole mechanism for extended
ops.</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005></SPAN></FONT> </DIV>
<DIV dir=ltr align=left><FONT face=Arial><FONT size=2><FONT
color=#0000ff><FONT><SPAN class=554400019-23122005>4. What is the purpose of
DAT_EXT_WRITE_CONFIRM_FLAG? Is it to expose IB round trip
semantic?</SPAN></FONT></FONT></FONT></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial><FONT size=2><FONT
color=#0000ff><FONT><SPAN class=554400019-23122005>iWARP does not support
immediate data. One can try to format payload to pass immediate
data.</SPAN></FONT></FONT></FONT></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial><FONT size=2><FONT
color=#0000ff><FONT><SPAN class=554400019-23122005>Is that what you had in
mind?</SPAN></FONT></FONT></FONT></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial><FONT size=2><FONT
color=#0000ff><FONT><SPAN
class=554400019-23122005></SPAN></FONT></FONT></FONT></FONT> </DIV>
<DIV dir=ltr align=left><FONT face=Arial><FONT size=2><FONT
color=#0000ff><FONT><SPAN class=554400019-23122005>What is the
sema</SPAN></FONT><FONT><SPAN class=554400019-23122005>ntic meaning of the
completion with this flag set? without flag
set?</SPAN></FONT></FONT></FONT></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>Are extended flags are additonal values for
COMPLETION_FLAGS? 2.4.1 talks about extended flags </SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>but where they are passed in is not
defined.</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>DAT 2.0 extended them already for FMR barrier. I would
prefer to follow that route rather than creating a separate</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>extension completion flags.</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005></SPAN></FONT> </DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>5. Why do you need RECV_IMMED? If Immed data is
delivered in event no new Recv operation is needed.</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>If Consumer asks for immediate data in payload where in
payload will it be?</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>If this is needed for local match for remote RDMA_Write
to handle immediate data lets state so.</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005></SPAN></FONT> </DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>What happens for mismatch between local and remote op?
That is recv was posted for Send and RDMA_Write</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>"arrived"? Vice Versa?</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005></SPAN></FONT><FONT face=Arial color=#0000ff
size=2><SPAN class=554400019-23122005></SPAN></FONT> </DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>6. I see extension for immediate data for rdma_write
but not for send. Is this deliberate? If we are going</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>to extend DAT semantic to support Immediate data we can
as well support the full IBTA/iWARP functionality for it.</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005></SPAN></FONT> </DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>7. Currently memory registration do not support access
to LMR or RMR by Atomic ops.</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>Do you propose to extend the meaning of current
MEM_PRIV for LMR and RMR to covers atomic accesses</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>or add new values to LMR_MEM_PRIV and RMR_MEM_PRIV for
atomic operation support?</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005></SPAN></FONT> </DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>8. Any alignment requirements for memory used for
atomic ops?</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005></SPAN></FONT> </DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>9. Any correlation requirements for SRQ buffers to
support recv with immediate data?</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005></SPAN></FONT> </DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>Have a great holidays,</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>Arkady</SPAN></FONT></DIV>
<DIV> </DIV><o:SmartTagType name="Street"
namespaceuri="urn:schemas-microsoft-com:office:smarttags"></o:SmartTagType><o:SmartTagType
name="address"
namespaceuri="urn:schemas-microsoft-com:office:smarttags"></o:SmartTagType>
<STYLE>st1\:* {
BEHAVIOR: url(#ieooui)
}
</STYLE>
<STYLE>@page Section1 {size: 8.5in 11.0in; margin: 1.0in 1.25in 1.0in 1.25in; mso-header-margin: .5in; mso-footer-margin: .5in; mso-paper-source: 0; }
P.MsoNormal {
FONT-SIZE: 12pt; MARGIN: 0in 0in 0pt; FONT-FAMILY: "Times New Roman"; mso-style-parent: ""; mso-pagination: widow-orphan; mso-fareast-font-family: "Times New Roman"
}
LI.MsoNormal {
FONT-SIZE: 12pt; MARGIN: 0in 0in 0pt; FONT-FAMILY: "Times New Roman"; mso-style-parent: ""; mso-pagination: widow-orphan; mso-fareast-font-family: "Times New Roman"
}
DIV.MsoNormal {
FONT-SIZE: 12pt; MARGIN: 0in 0in 0pt; FONT-FAMILY: "Times New Roman"; mso-style-parent: ""; mso-pagination: widow-orphan; mso-fareast-font-family: "Times New Roman"
}
A:link {
COLOR: blue; TEXT-DECORATION: underline; text-underline: single
}
SPAN.MsoHyperlink {
COLOR: blue; TEXT-DECORATION: underline; text-underline: single
}
A:visited {
COLOR: purple; TEXT-DECORATION: underline; text-underline: single
}
SPAN.MsoHyperlinkFollowed {
COLOR: purple; TEXT-DECORATION: underline; text-underline: single
}
SPAN.SpellE {
mso-style-name: ""; mso-spl-e: yes
}
SPAN.GramE {
mso-style-name: ""; mso-gram-e: yes
}
DIV.Section1 {
page: Section1
}
</STYLE>
<DIV class=Section1>
<P class=MsoNormal align=left><SPAN class=SpellE><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Arkady</SPAN></SPAN><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: Arial"> <SPAN
class=SpellE>Kanevsky</SPAN><SPAN
style="mso-tab-count: 2">
</SPAN>email: <A
href="mailto:arkady@netapp.com">arkady@netapp.com</A><o:p></o:p></SPAN></P>
<P class=MsoNormal><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Network
Appliance Inc.<SPAN
style="mso-tab-count: 2">
</SPAN><SPAN class=GramE>phone</SPAN>: 781-768-5395<o:p></o:p></SPAN></P>
<P class=MsoNormal><st1:Street w:st="on"><st1:address
style="BACKGROUND-POSITION: left bottom; BACKGROUND-IMAGE: url(res://ietag.dll/#34/#1001); BACKGROUND-REPEAT: repeat-x"
tabIndex=0 w:st="on"><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">1601
Trapelo Rd. - Suite 16.</SPAN></st1:address></st1:Street><SPAN
style="mso-tab-count: 2"> </SPAN><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Fax:
781-895-1195<o:p></o:p></SPAN></P>
<P class=MsoNormal><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Waltham, MA
02451
central phone: 781-768-5300</SPAN></P></DIV>
<DIV> </DIV><BR>
<BLOCKQUOTE dir=ltr
style="PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT: #0000ff 2px solid; MARGIN-RIGHT: 0px">
<DIV class=OutlookMessageHeader lang=en-us dir=ltr align=left>
<HR tabIndex=-1>
<FONT face=Tahoma size=2><B>From:</B> Arlin Davis
[mailto:arlin.r.davis@intel.com] <BR><B>Sent:</B> Thursday, December 22, 2005
6:20 PM<BR><B>To:</B> Lentini, James; Kanevsky, Arkady<BR><B>Cc:</B>
openib-general@openib.org; dat-discussions@yahoogroups.com<BR><B>Subject:</B>
[RFC][PATCH] OpenIB uDAPL extension proposal - sample immed data and atomic
api's<BR></FONT><BR></DIV>
<DIV></DIV>
<DIV class=Section1>
<P class=MsoNormal><FONT face="Times New Roman" size=2><SPAN
style="FONT-SIZE: 11pt">James and Arkady,<o:p></o:p></SPAN></FONT></P>
<P class=MsoNormal><FONT face="Times New Roman"
size=2><o:p> </o:p></FONT></P>
<P class=MsoNormal><FONT face=Arial size=2><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">DAPL provides a generalized
abstraction to RDMA capable transports. As a generalized abstraction, it
cannot exploit the unique properties that many of the underlying
platforms/interconnects can provide so I would like to propose a simple
(minimum impact on libdat) extensible interface to uDAPL that will allow
vendors to expose such capabilities. I am looking for feedback, especially
from the DAT collaborative. I have included both a design document and actual
working code as a reference.<o:p></o:p></SPAN></FONT></P>
<P class=MsoNormal><FONT face=Arial size=2><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: Arial"><o:p> </o:p></SPAN></FONT></P>
<P class=MsoNormal><FONT face=Arial size=2><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">The patch provides a fully tested
DAT and DAPL library (openib_cma) set with the following provider
extensions:<o:p></o:p></SPAN></FONT></P>
<P class=MsoNormal><FONT face=Arial size=2><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: Arial"><o:p> </o:p></SPAN></FONT></P>
<P class=MsoNormal><FONT face=Courier size=2><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: Courier">DAT_RETURN<BR>dat_ep_post_write_immed(<BR>
IN DAT_EP_HANDLE
ep_handle,<o:p></o:p></SPAN></FONT></P>
<P class=MsoNormal
style="MARGIN-BOTTOM: 0pt; MARGIN-LEFT: 0.5in; MARGIN-RIGHT: 0in; mso-margin-top-alt: 0in"><FONT
face=Courier size=2><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Courier">IN
DAT_COUNT
num_segments<BR>IN DAT_LMR_TRIPLET
*local_iov,<BR>IN DAT_DTO_COOKIE
user_cookie,<o:p></o:p></SPAN></FONT></P>
<P class=MsoNormal
style="MARGIN-BOTTOM: 0pt; MARGIN-LEFT: 0.5in; MARGIN-RIGHT: 0in; mso-margin-top-alt: 0in"><FONT
face=Courier size=2><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Courier">IN
DAT_RMR_TRIPLE *remote_iov,<BR>IN
DAT_UINT32
immediate_data,<BR>IN DAT_COMPLETION_FLAGS
completion_flags);<o:p></o:p></SPAN></FONT></P>
<P class=MsoNormal><FONT face=Arial size=2><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: Arial"><o:p> </o:p></SPAN></FONT></P>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt"><FONT face=Courier size=2><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: Courier">DAT_RETURN<BR>dat_ep_post_cmp_and_swap(<BR>
IN DAT_EP_HANDLE
ep_handle,<o:p></o:p></SPAN></FONT></P>
<P class=MsoNormal
style="MARGIN-BOTTOM: 0pt; MARGIN-LEFT: 0.5in; MARGIN-RIGHT: 0in; mso-margin-top-alt: 0in"><FONT
face=Courier size=2><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Courier">IN
DAT_UINT64
cmp_value,<BR>IN
DAT_UINT64
swap_value,<o:p></o:p></SPAN></FONT></P>
<P class=MsoNormal
style="MARGIN-BOTTOM: 0pt; MARGIN-LEFT: 0.5in; MARGIN-RIGHT: 0in; mso-margin-top-alt: 0in"><FONT
face=Courier size=2><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Courier">IN
DAT_LMR_TRIPLE *local_iov,<BR>IN
DAT_DTO_COOKIE
user_cookie,<o:p></o:p></SPAN></FONT></P>
<P class=MsoNormal
style="MARGIN-BOTTOM: 0pt; MARGIN-LEFT: 0.5in; MARGIN-RIGHT: 0in; mso-margin-top-alt: 0in"><FONT
face=Courier size=2><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Courier">IN
DAT_RMR_TRIPLE *remote_iov,<BR>IN
DAT_COMPLETION_FLAGS completion_flags);<o:p></o:p></SPAN></FONT></P>
<P class=MsoNormal><FONT face=Arial size=2><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: Arial"><o:p> </o:p></SPAN></FONT></P>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt"><FONT face=Courier size=2><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: Courier">DAT_RETURN<BR>dat_ep_post_fetch_and_add(<BR>
IN DAT_EP_HANDLE
ep_handle,<o:p></o:p></SPAN></FONT></P>
<P class=MsoNormal
style="MARGIN-BOTTOM: 0pt; MARGIN-LEFT: 0.5in; MARGIN-RIGHT: 0in; mso-margin-top-alt: 0in"><FONT
face=Courier size=2><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Courier">IN
DAT_UINT64
add_value,<o:p></o:p></SPAN></FONT></P>
<P class=MsoNormal
style="MARGIN-BOTTOM: 0pt; MARGIN-LEFT: 0.5in; MARGIN-RIGHT: 0in; mso-margin-top-alt: 0in"><FONT
face=Courier size=2><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Courier">IN
DAT_LMR_TRIPLE *local_iov,<BR>IN
DAT_DTO_COOKIE
user_cookie,<o:p></o:p></SPAN></FONT></P>
<P class=MsoNormal
style="MARGIN-BOTTOM: 0pt; MARGIN-LEFT: 0.5in; MARGIN-RIGHT: 0in; mso-margin-top-alt: 0in"><FONT
face=Courier size=2><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Courier">IN
DAT_RMR_TRIPLE *remote_iov,<BR>IN
DAT_COMPLETION_FLAGS completion_flags);<o:p></o:p></SPAN></FONT></P>
<P class=MsoNormal><FONT face=Arial size=2><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: Arial"><o:p> </o:p></SPAN></FONT></P>
<P class=MsoNormal><FONT face=Arial size=2><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Also, included is a sample program
(dtest_ext.c) that can be used as a programming
example.<o:p></o:p></SPAN></FONT></P>
<P class=MsoNormal><FONT face=Arial size=2><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: Arial"><o:p> </o:p></SPAN></FONT></P>
<P class=MsoNormal><FONT face=Arial size=2><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Thanks,<o:p></o:p></SPAN></FONT></P>
<P class=MsoNormal><FONT face=Arial size=2><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: Arial"><o:p> </o:p></SPAN></FONT></P>
<P class=MsoNormal><FONT face=Arial size=2><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">-arlin<o:p></o:p></SPAN></FONT></P>
<P class=MsoNormal><FONT face=Arial size=2><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: Arial"><o:p> </o:p></SPAN></FONT></P><PRE><FONT face="Courier New" size=2><SPAN style="FONT-SIZE: 10pt">Signed-off by: Arlin Davis <A href="mailto:ardavis@ichips.intel.com"><ardavis@ichips.intel.com></A><o:p></o:p></SPAN></FONT></PRE><PRE><FONT face="Courier New" size=2><SPAN style="FONT-SIZE: 10pt"><o:p> </o:p></SPAN></FONT></PRE><PRE><FONT face="Courier New" size=2><SPAN style="FONT-SIZE: 10pt"><o:p> </o:p></SPAN></FONT></PRE><PRE><FONT face="Courier New" size=2><SPAN style="FONT-SIZE: 10pt"><o:p> </o:p></SPAN></FONT></PRE>
<P class=MsoPlainText><FONT face="Courier New" size=2><SPAN
style="FONT-SIZE: 10pt">Index:
test/dtest/dat.conf<BR>===================================================================<BR>---
test/dtest/dat.conf (revision 4589)<BR>+++ test/dtest/dat.conf (working
copy)<BR>@@ -1,11 +1,20 @@<BR> #<BR>-# DAT 1.1 and 1.2 configuration
file<BR>+# DAT 1.2 configuration file<BR> #<BR> # Each entry should
have the following fields:<BR> #<BR> # <ia_name>
<api_version> <threadsafety> <default> <lib_path>
\<BR> #
<provider_version> <ia_params>
<platform_params><BR> #<BR>-# Example for openib using the first
Mellanox adapter, port 1 and port 2<BR>+# Example for openib_cma and
openib_scm<BR>+#<BR>+# For scm version you specify <ia_params> as actual
device name and port<BR>+# For cma version you specify <ia_params>
as:<BR>+# network address, network
hostname, or netdev name and 0 for port<BR>+#<BR>+OpenIB-scm1 u1.2
nonthreadsafe default /usr/local/openib_dapl/udapl/Target/libdapl.so
mv_dapl.1.2 "mthca0 1" ""<BR>+OpenIB-scm2 u1.2 nonthreadsafe default
/usr/local/openib_dapl/udapl/Target/libdapl.so mv_dapl.1.2 "mthca0 2"
""<BR>+OpenIB-cma-ip u1.2 nonthreadsafe default
/usr/local/openib_dapl/udapl/Target/libdapl.so mv_dapl.1.2 "192.168.0.22 0"
""<BR>+OpenIB-cma-name u1.2 nonthreadsafe default
/usr/local/openib_dapl/udapl/Target/libdapl.so mv_dapl.1.2 "svr1-ib0 0"
""<BR>+OpenIB-cma-netdev u1.2 nonthreadsafe default
/usr/local/openib_dapl/udapl/Target/libdapl.so mv_dapl.1.2 "ib0 0"
""<BR> <BR>-IB1 u1.2 nonthreadsafe default <BR>Index:
test/dtest/makefile<BR>===================================================================<BR>---
test/dtest/makefile (revision 4589)<BR>+++ test/dtest/makefile (working
copy)<BR>@@ -4,13 +4,18 @@ CFLAGS = -O2
-g<BR> DAT_INC = ../../dat/include<BR> DAT_LIB =
/usr/local/lib<BR> <BR>-all: dtest<BR>+all: dtest
dtest_ext<BR> <BR> clean:<BR>- rm -f
*.o;touch *.c;rm -f dtest<BR>+ rm -f *.o;touch *.c;rm
-f dtest dtest_ext<BR> <BR> dtest:
./dtest.c<BR> $(CC) $(CFLAGS) ./dtest.c -o dtest
\<BR> -DDAPL_PROVIDER='"OpenIB-cma-ip"'
\<BR> -I $(DAT_INC) -L $(DAT_LIB)
-ldat<BR> <BR>+dtest_ext: ./dtest_ext.c<BR>+
$(CC) $(CFLAGS) ./dtest_ext.c -o dtest_ext \<BR>+
-DDAPL_PROVIDER='"OpenIB-cma-ip"' \<BR>+ -I $(DAT_INC)
-L $(DAT_LIB) -ldat<BR>+<BR>Index:
test/dtest/README<BR>===================================================================<BR>---
test/dtest/README (revision 4589)<BR>+++
test/dtest/README (working copy)<BR>@@ -1,10 +1,11
@@<BR> simple dapl test just for initial openIB uDAPL
testing...<BR> <BR>
dtest/dtest.c<BR>+
dtest/dtest_ext.c<BR>
dtest/makefile<BR>
dtest/dat.conf<BR> <BR>-to build (default uDAPL name == IB1, ib
device == mthca0, port == 1)<BR>+to build (default uDAPL name ==
OpenIB-cma-ip)<BR> edit makefile and
change path (DAT_LIB) to appropriate
libdat.so<BR> edit dat.conf and
change path to appropriate
libdapl.so<BR> cp dat.conf to
/etc/dat.conf<BR>Index:
dapl/include/dapl.h<BR>===================================================================<BR>---
dapl/include/dapl.h (revision 4589)<BR>+++ dapl/include/dapl.h (working
copy)<BR>@@ -1,25 +1,28 @@<BR> /*<BR>- * Copyright (c) 2002-2003, Network
Appliance, Inc. All rights reserved.<BR>+ * Copyright (c) 2002-2005, Network
Appliance, Inc. All rights reserved.<BR> *<BR> * This Software is
licensed under one of the following licenses:<BR>- *<BR>+ * <BR> * 1)
under the terms of the "Common Public License 1.0" a copy of which is<BR>-
* available from the Open Source Initiative, see<BR>+
* in the file LICENSE.txt in the root directory. The license
is also<BR>+ * available from the Open Source Initiative,
see <BR> *
http://www.opensource.org/licenses/cpl.php.<BR>- *<BR>- * 2) under the terms
of the "The BSD License" a copy of which is<BR>- * available
from the Open Source Initiative, see<BR>+ * <BR>+ * 2) under the terms of the
"The BSD License" a copy of which is in the file<BR>+ *
LICENSE2.txt in the root directory. The license is also available from<BR>+
* the Open Source Initiative, see<BR>
* http://www.opensource.org/licenses/bsd-license.php.<BR>-
*<BR>+ * <BR> * 3) under the terms of the "GNU General Public License
(GPL) Version 2" a<BR>- * copy of which is available from
the Open Source Initiative, see<BR>+ * copy of which is in
the file LICENSE3.txt in the root directory. The<BR>+ *
license is also available from the Open Source Initiative, see<BR>
* http://www.opensource.org/licenses/gpl-license.php.<BR>-
*<BR>+ * <BR> * Licensee has the right to choose one of the above
licenses.<BR>- *<BR>+ * <BR> * Redistributions of source code must
retain the above copyright<BR> * notice and one of the license
notices.<BR>- *<BR>+ * <BR> * Redistributions in binary form must
reproduce both the above copyright<BR> * notice, one of the license
notices in the documentation<BR> * and/or other materials provided with
the distribution.<BR>@@ -61,6 +64,8 @@<BR> #include
"dapl_dummy_util.h"<BR> #elif OPENIB<BR> #include
"dapl_ib_util.h"<BR>+#elif DET<BR>+#include
"dapl_det_util.h"<BR> #endif<BR> <BR> /*********************************************************************<BR>@@
-213,6 +218,10 @@ typedef struct
dapl_cookie DAPL_COOKIE;<BR> typedef
struct
dapl_dto_cookie
DAPL_DTO_COOKIE;<BR> typedef struct
dapl_rmr_cookie
DAPL_RMR_COOKIE;<BR> <BR>+#ifdef DAPL_EXTENSIONS<BR>+typedef struct
dapl_ext_cookie
DAPL_EXT_COOKIE;<BR>+#endif<BR>+<BR> typedef struct
dapl_private
DAPL_PRIVATE;<BR> <BR> typedef void (*DAPL_CONNECTION_STATE_HANDLER)
(<BR>@@ -563,6 +572,13 @@ typedef enum
dapl_dto_type<BR>
DAPL_DTO_TYPE_RECV,<BR>
DAPL_DTO_TYPE_RDMA_WRITE,<BR>
DAPL_DTO_TYPE_RDMA_READ,<BR>+#ifdef DAPL_EXTENSIONS<BR>+
DAPL_DTO_TYPE_RDMA_WRITE_IMMED,<BR>+
DAPL_DTO_TYPE_RECV_IMMED,<BR>+
DAPL_DTO_TYPE_CMP_AND_SWAP,<BR>+
DAPL_DTO_TYPE_FETCH_AND_ADD,<BR>+#endif<BR>+<BR> }
DAPL_DTO_TYPE;<BR> <BR> typedef enum dapl_cookie_type<BR>@@ -570,6
+586,9 @@ typedef enum dapl_cookie_type<BR>
DAPL_COOKIE_TYPE_NULL,<BR>
DAPL_COOKIE_TYPE_DTO,<BR>
DAPL_COOKIE_TYPE_RMR,<BR>+#ifdef DAPL_EXTENSIONS<BR>+
DAPL_COOKIE_TYPE_EXTENSION,<BR>+#endif<BR> }
DAPL_COOKIE_TYPE;<BR> <BR> /* DAPL_DTO_COOKIE used as context for
DTO WQEs */<BR>@@ -587,6 +606,27 @@ struct
dapl_rmr_cookie<BR>
DAT_RMR_COOKIE
cookie;<BR> };<BR> <BR>+#ifdef DAPL_EXTENSIONS<BR>+<BR>+/* DAPL
extended cookie types */<BR>+typedef enum
dapl_ext_type<BR>+{<BR>+
DAPL_EXT_TYPE_RDMA_WRITE_IMMED,<BR>+
DAPL_EXT_TYPE_CMP_AND_SWAP,<BR>+
DAPL_EXT_TYPE_FETCH_AND_ADD,<BR>+ DAPL_EXT_TYPE_RECV<BR>+}
DAPL_EXT_TYPE;<BR>+<BR>+/* DAPL extended cookie */<BR>+struct
dapl_ext_cookie<BR>+{<BR>+
DAPL_EXT_TYPE
type;<BR>+
DAT_DTO_COOKIE
cookie;<BR>+
DAT_COUNT
size; /* used RDMA write with immed */<BR>+};<BR>+<BR>+#endif<BR>+<BR> /*
DAPL_COOKIE used as context for WQEs */<BR> struct
dapl_cookie<BR> {<BR>@@ -597,6 +637,9 @@ struct
dapl_cookie<BR> {<BR>
DAPL_DTO_COOKIE dto;<BR>
DAPL_RMR_COOKIE
rmr;<BR>+#ifdef DAPL_EXTENSIONS<BR>+
DAPL_EXT_COOKIE
ext;<BR>+#endif<BR> } val;<BR> };<BR> <BR>@@
-1116,6 +1159,15 @@ extern DAT_RETURN
dapl_srq_set_lw(<BR>
IN
DAT_SRQ_HANDLE, /*
srq_handle
*/<BR> IN
DAT_COUNT);
/* low_watermark
*/<BR> <BR>+#ifdef DAPL_EXTENSIONS<BR>+<BR>+extern DAT_RETURN
dapl_extensions(<BR>+ IN
DAT_HANDLE, /*
dat_handle
*/<BR>+ IN DAT_EXT_OP,
/*
extension operation */<BR>+ IN
va_list ); /* va_list
args
*/<BR>+<BR>+#endif<BR>+<BR> /*<BR> * DAPL internal utility function
prototpyes<BR> */<BR>Index:
dapl/udapl/Makefile<BR>===================================================================<BR>---
dapl/udapl/Makefile (revision 4589)<BR>+++ dapl/udapl/Makefile (working
copy)<BR>@@ -156,6 +156,7 @@ PROVIDER =
$(TOPDIR)/../openib_cma<BR> CFLAGS +=
-DOPENIB<BR> CFLAGS +=
-DCQ_WAIT_OBJECT<BR> CFLAGS +=
-I/usr/local/include/infiniband<BR>+CFLAGS +=
-I/usr/local/include/rdma<BR> endif<BR> <BR> #<BR>@@ -168,6
+169,12 @@ endif<BR> # VN_MEM_SHARED_VIRTUAL_SUPPORT <BR> #
CFLAGS += -DVN_MEM_SHARED_VIRTUAL_SUPPORT=1<BR> <BR>+# If an
implementation supports DAPL extensions<BR>+CFLAGS +=
-DDAPL_EXTENSIONS<BR>+<BR>+# If an implementation supports DAPL provider
specific attributes<BR>+CFLAGS +=
-DDAPL_PROVIDER_SPECIFIC_ATTR<BR>+<BR> CFLAGS += -I.
<BR> CFLAGS += -I.. <BR> CFLAGS +=
-I../../dat/include <BR>@@ -283,6 +290,8 @@ LDFLAGS += -libverbs
-lrdmacm<BR> LDFLAGS += -rpath /usr/local/lib -L
/usr/local/lib<BR> PROVIDER_SRCS = dapl_ib_util.c dapl_ib_cq.c
dapl_ib_qp.c
\<BR>
dapl_ib_cm.c dapl_ib_mem.c<BR>+# implementation supports DAPL
extensions<BR>+PROVIDER_SRCS +=
dapl_ib_extensions.c<BR> endif<BR> <BR> UDAPL_SRCS =
dapl_init.c \<BR>Index:
dapl/common/dapl_ia_query.c<BR>===================================================================<BR>---
dapl/common/dapl_ia_query.c (revision 4589)<BR>+++
dapl/common/dapl_ia_query.c (working copy)<BR>@@
-167,6 +167,14 @@ dapl_ia_query (<BR> #if
!defined(__KDAPL__)<BR>
provider_attr->pz_support
= DAT_PZ_UNIQUE;<BR> #endif /* !KDAPL
*/<BR>+<BR>+ /*<BR>+ *
Have provider set their own.<BR>+ */<BR>+#ifdef
DAPL_PROVIDER_SPECIFIC_ATTR<BR>+
dapls_set_provider_specific_attr(provider_attr);<BR>+#endif<BR>+<BR>
/*<BR> * Set up
evd_stream_merging_supported options. Note there
is<BR> * one bit per allowable
combination, using the ordinal<BR>Index:
dapl/common/dapl_adapter_util.h<BR>===================================================================<BR>---
dapl/common/dapl_adapter_util.h (revision 4589)<BR>+++
dapl/common/dapl_adapter_util.h (working copy)<BR>@@ -256,6 +256,21 @@
dapls_ib_wait_object_wait (<BR> IN u_int32_t
timeout);<BR> #endif<BR> <BR>+#ifdef
DAPL_PROVIDER_SPECIFIC_ATTR<BR>+void
<BR>+dapls_set_provider_specific_attr(<BR>+ IN
DAT_PROVIDER_ATTR
*provider_attr );<BR>+#endif<BR>+<BR>+#ifdef
DAPL_EXTENSIONS<BR>+void<BR>+dapls_cqe_to_event_extension(<BR>+
IN
DAPL_EP
*ep_ptr,<BR>+ IN
DAPL_COOKIE
*cookie,<BR>+ IN
ib_work_completion_t
*cqe_ptr,<BR>+ OUT
DAT_EVENT
*event_ptr);<BR>+#endif<BR>+<BR> /*<BR> * Values for provider
DAT_NAMED_ATTR<BR> */<BR>@@ -272,6 +287,8 @@ dapls_ib_wait_object_wait
(<BR> #include "dapl_dummy_dto.h"<BR> #elif OPENIB<BR> #include
"dapl_ib_dto.h"<BR>+#elif DET<BR>+#include
"dapl_det_dto.h"<BR> #endif<BR> <BR> <BR>Index:
dapl/common/dapl_provider.c<BR>===================================================================<BR>---
dapl/common/dapl_provider.c (revision 4589)<BR>+++
dapl/common/dapl_provider.c (working copy)<BR>@@
-221,7 +221,11 @@ DAT_PROVIDER g_dapl_provider_template
=<BR>
&dapl_srq_post_recv,<BR>
&dapl_srq_query,<BR>
&dapl_srq_resize,<BR>-
&dapl_srq_set_lw<BR>+
&dapl_srq_set_lw,<BR>+<BR>+#ifdef DAPL_EXTENSIONS<BR>+
&dapl_extensions<BR>+#endif<BR> <BR> };<BR> #endif /*
__KDAPL__ */<BR>Index:
dapl/common/dapl_evd_util.c<BR>===================================================================<BR>---
dapl/common/dapl_evd_util.c (revision 4589)<BR>+++
dapl/common/dapl_evd_util.c (working copy)<BR>@@
-502,6 +502,20 @@ dapli_evd_eh_print_cqe (<BR> #ifdef
DAPL_DBG<BR> static char *optable[]
=<BR> {<BR>+#ifdef OPENIB<BR>+
/* different order for openib verbs */<BR>+
"OP_RDMA_WRITE",<BR>+
"OP_RDMA_WRITE_IMM",<BR>+
"OP_SEND",<BR>+
"OP_SEND_IMM",<BR>+
"OP_RDMA_READ",<BR>+
"OP_COMP_AND_SWAP",<BR>+
"OP_FETCH_AND_ADD",<BR>+
"OP_RECEIVE",<BR>+
"OP_RECEIVE_IMM",<BR>+
"OP_BIND_MW",<BR>+
"OP_INVALID",<BR>+#else<BR>
"OP_SEND",<BR>
"OP_RDMA_READ",<BR> "OP_RDMA_WRITE",<BR>@@
-509,6 +523,7 @@ dapli_evd_eh_print_cqe (<BR>
"OP_FETCH_AND_ADD",<BR>
"OP_RECEIVE",<BR>
"OP_BIND_MW",<BR>+#endif<BR>
0<BR> };<BR> <BR>@@ -1113,6 +1128,15 @@
dapli_evd_cqe_to_event (<BR>
dapls_cookie_dealloc (&ep_ptr->req_buffer,
cookie);<BR>
break;<BR> }<BR>+<BR>+#ifdef
DAPL_EXTENSIONS<BR>+ case
DAPL_COOKIE_TYPE_EXTENSION:<BR>+
{<BR>+
dapls_cqe_to_event_extension(ep_ptr, cookie, cqe_ptr,
event_ptr);<BR>+
break;<BR>+
}<BR>+#endif<BR>+<BR>
default:<BR> {<BR>
dapl_os_assert (!"Invalid Operation type");<BR>Index:
dapl/openib_cma/dapl_ib_dto.h<BR>===================================================================<BR>---
dapl/openib_cma/dapl_ib_dto.h (revision 4589)<BR>+++
dapl/openib_cma/dapl_ib_dto.h (working copy)<BR>@@ -35,7 +35,7
@@<BR> *<BR> * Description: <BR> *<BR>-
* The uDAPL openib provider - DTO operations and CQE macros <BR>+
* The OpenIB uCMA provider - DTO operations and CQE macros
<BR> *<BR>
****************************************************************************<BR>
* Source Control
System Information<BR>@@ -119,7 +119,6 @@ dapls_ib_post_recv
(<BR> return
DAT_SUCCESS;<BR> }<BR> <BR>-<BR> /*<BR> *
dapls_ib_post_send<BR> *<BR>@@ -191,7 +190,7 @@ dapls_ib_post_send
(<BR> <BR> if (cookie != NULL)
<BR>
cookie->val.dto.size = total_len;<BR>-
<BR>+<BR> if ((op_type == OP_RDMA_WRITE) ||
(op_type == OP_RDMA_READ))
{<BR>
wr.wr.rdma.remote_addr =
remote_iov->target_address;<BR>
wr.wr.rdma.rkey = remote_iov->rmr_context;<BR>@@ -224,6 +223,152 @@
dapls_ib_post_send (<BR> return
DAT_SUCCESS;<BR> }<BR> <BR>+#ifdef DAPL_EXTENSIONS<BR>+/*<BR>+ *
dapls_ib_post_ext_send<BR>+ *<BR>+ * Provider specific extended Post SEND
function<BR>+ */<BR>+STATIC _INLINE_ DAT_RETURN <BR>+dapls_ib_post_ext_send
(<BR>+ IN
DAPL_EP
*ep_ptr,<BR>+ IN
ib_send_op_type_t
op_type,<BR>+ IN
DAPL_COOKIE
*cookie,<BR>+ IN
DAT_COUNT
segments,<BR>+ IN
DAT_LMR_TRIPLET
*local_iov,<BR>+ IN const
DAT_RMR_TRIPLET
*remote_iov,<BR>+ IN
DAT_UINT32
idata,<BR>+ IN
DAT_UINT64
compare_add,<BR>+ IN
DAT_UINT64
swap,<BR>+ IN
DAT_COMPLETION_FLAGS
completion_flags)<BR>+{<BR>+
dapl_dbg_log(DAPL_DBG_TYPE_EP,<BR>+
" post_snd: ep %p op %d ck %p
sgs",<BR>+
"%d l_iov %p r_iov %p f
%d\n",<BR>+
ep_ptr, op_type, cookie, segments, local_iov,
<BR>+
remote_iov,
completion_flags);<BR>+<BR>+ ib_data_segment_t
ds_array[DEFAULT_DS_ENTRIES];<BR>+ ib_data_segment_t
*ds_array_p;<BR>+ struct ibv_send_wr
wr;<BR>+ struct ibv_send_wr
*bad_wr;<BR>+ ib_hca_transport_t *ibt_ptr =
<BR>+
&ep_ptr->header.owner_ia->hca_ptr->ib_trans;<BR>+
DAT_COUNT i, total_len;<BR>+
<BR>+
dapl_dbg_log(DAPL_DBG_TYPE_EP,<BR>+
" post_snd: ep %p cookie %p segs %d l_iov
%p\n",<BR>+
ep_ptr, cookie, segments,
local_iov);<BR>+<BR>+ if(segments <=
DEFAULT_DS_ENTRIES)
<BR>+ ds_array_p =
ds_array;<BR>+
else<BR>+
ds_array_p =
<BR>+
dapl_os_alloc(segments *
sizeof(ib_data_segment_t));<BR>+<BR>+ if (NULL ==
ds_array_p)<BR>+
return (DAT_INSUFFICIENT_RESOURCES);<BR>+
<BR>+ /* setup the work request
*/<BR>+ wr.next = 0;<BR>+
wr.opcode = op_type;<BR>+ wr.num_sge =
0;<BR>+ wr.send_flags =
0;<BR>+ wr.wr_id =
(uint64_t)(uintptr_t)cookie;<BR>+ wr.sg_list =
ds_array_p;<BR>+ total_len =
0;<BR>+<BR>+ for (i = 0; i < segments; i++ )
{<BR>+ if (
!local_iov[i].segment_length
)<BR>+
continue;<BR>+<BR>+
ds_array_p->addr = (uint64_t)
local_iov[i].virtual_address;<BR>+
ds_array_p->length =
local_iov[i].segment_length;<BR>+
ds_array_p->lkey =
local_iov[i].lmr_context;<BR>+
<BR>+
dapl_dbg_log(DAPL_DBG_TYPE_EP,
<BR>+
" post_snd: lkey 0x%x va %p len
%d\n",<BR>+
ds_array_p->lkey, ds_array_p->addr,
<BR>+
ds_array_p->length
);<BR>+<BR>+
total_len +=
ds_array_p->length;<BR>+
wr.num_sge++;<BR>+
ds_array_p++;<BR>+ }<BR>+<BR>+
if (cookie != NULL)
<BR>+
cookie->val.dto.size = total_len;<BR>+<BR>+ if
((op_type == OP_RDMA_WRITE) || <BR>+
(op_type == OP_RDMA_WRITE_IMM)
||<BR>+ (op_type == OP_RDMA_READ))
{<BR>+
wr.wr.rdma.remote_addr =
remote_iov->target_address;<BR>+
wr.wr.rdma.rkey =
remote_iov->rmr_context;<BR>+
dapl_dbg_log(DAPL_DBG_TYPE_EP,
<BR>+
" post_snd_rdma: rkey 0x%x va
%#016Lx\n",<BR>+
wr.wr.rdma.rkey,
wr.wr.rdma.remote_addr);<BR>+
}<BR>+<BR>+ switch (op_type)
{<BR>+ case
OP_RDMA_WRITE_IMM:<BR>+
dapl_dbg_log(DAPL_DBG_TYPE_EP,
<BR>+
" post_snd: OP_RDMA_WRITE_IMMED=0x%x\n", idata
);<BR>+
wr.imm_data =
idata;<BR>+
break;<BR>+ case
OP_COMP_AND_SWAP:<BR>+
/* OP_COMP_AND_SWAP has direct IBAL wr_type mapping
*/<BR>+
dapl_dbg_log(DAPL_DBG_TYPE_EP,
<BR>+
" post_snd:
OP_COMP_AND_SWAP=%lx,"<BR>+
"%lx rkey 0x%x va
%#016Lx\n",<BR>+
compare_add, swap,
remote_iov->rmr_context,<BR>+
remote_iov->target_address);<BR>+
<BR>+
wr.wr.atomic.compare_add =
compare_add;<BR>+
wr.wr.atomic.swap =
swap;<BR>+
wr.wr.atomic.remote_addr =
remote_iov->target_address;<BR>+
wr.wr.atomic.rkey =
remote_iov->rmr_context;<BR>+
break;<BR>+ case
OP_FETCH_AND_ADD:<BR>+
/* OP_FETCH_AND_ADD has direct IBAL wr_type mapping
*/<BR>+
dapl_dbg_log(DAPL_DBG_TYPE_EP,
<BR>+
" post_snd:
OP_FETCH_AND_ADD=%lx,"<BR>+
"%lx rkey 0x%x va
%#016Lx\n",<BR>+
compare_add,
remote_iov->rmr_context,<BR>+
remote_iov->target_address);<BR>+<BR>+
wr.wr.atomic.compare_add =
compare_add;<BR>+
wr.wr.atomic.remote_addr =
remote_iov->target_address;<BR>+
wr.wr.atomic.rkey =
remote_iov->rmr_context;<BR>+
break;<BR>+
default:<BR>+
break;<BR>+ }<BR>+<BR>+ /*
inline data for send or write ops */<BR>+ if
((total_len <= ibt_ptr->max_inline_send) &&
<BR>+ ((op_type == OP_SEND) || (op_type
== OP_RDMA_WRITE)))
<BR>+
wr.send_flags |= IBV_SEND_INLINE;<BR>+
<BR>+ /* set completion flags in work request
*/<BR>+ wr.send_flags |= (DAT_COMPLETION_SUPPRESS_FLAG
&
<BR>+
completion_flags) ? 0 : IBV_SEND_SIGNALED;<BR>+
wr.send_flags |= (DAT_COMPLETION_BARRIER_FENCE_FLAG &
<BR>+
completion_flags) ? IBV_SEND_FENCE : 0;<BR>+
wr.send_flags |= (DAT_COMPLETION_SOLICITED_WAIT_FLAG &
<BR>+
completion_flags) ? IBV_SEND_SOLICITED : 0;<BR>+<BR>+
dapl_dbg_log(DAPL_DBG_TYPE_EP,
<BR>+
" post_snd: op 0x%x flags 0x%x sglist %p, %d\n",
<BR>+
wr.opcode, wr.send_flags, wr.sg_list,
wr.num_sge);<BR>+<BR>+ if
(ibv_post_send(ep_ptr->qp_handle->cm_id->qp, &wr,
&bad_wr))<BR>+
return( dapl_convert_errno(EFAULT,"ibv_recv") );<BR>+
<BR>+ dapl_dbg_log(DAPL_DBG_TYPE_EP," post_snd:
returned\n");<BR>+ return
DAT_SUCCESS;<BR>+}<BR>+#endif<BR>+<BR> STATIC _INLINE_ DAT_RETURN
<BR> dapls_ib_optional_prv_dat(<BR>
IN DAPL_CR *cr_ptr,<BR>Index:
dapl/openib_cma/dapl_ib_util.c<BR>===================================================================<BR>---
dapl/openib_cma/dapl_ib_util.c (revision 4589)<BR>+++
dapl/openib_cma/dapl_ib_util.c (working copy)<BR>@@ -35,7 +35,7
@@<BR> *<BR> * Description: <BR> *<BR>-
* The uDAPL openib provider - init, open, close, utilities, work
thread<BR>+ * The OpenIB uCMA provider - init, open, close,
utilities, work thread<BR> *<BR>
****************************************************************************<BR>
* Source Control
System Information<BR>@@ -64,7 +64,6 @@ static const char rcsid[] =
"$Id: $";<BR> #include <net/if.h> /*
for struct ifreq */<BR> #include <net/if_arp.h> /* for
ARPHRD_INFINIBAND */<BR> <BR>-<BR> int g_dapl_loopback_connection =
0;<BR> int g_ib_pipe[2];<BR> ib_thread_state_t g_ib_thread_state =
0;<BR>@@ -727,7 +726,7 @@ void dapli_thread(void *arg)
<BR>
int
ret,idx,fds;<BR>
char
rbuf[2];<BR> <BR>-
dapl_dbg_log (DAPL_DBG_TYPE_CM,<BR>+ dapl_dbg_log
(DAPL_DBG_TYPE_UTIL,<BR>
" ib_thread(%d,0x%x): ENTER: pipe %d ucma
%d\n",<BR>
getpid(), g_ib_thread, g_ib_pipe[0],
rdma_get_fd());<BR> <BR>@@ -767,7 +766,7 @@ void dapli_thread(void *arg)
<BR>
ufds[idx].revents = 0;<BR>
uhca[idx] =
hca;<BR> <BR>-
dapl_dbg_log(DAPL_DBG_TYPE_CM,<BR>+
dapl_dbg_log(DAPL_DBG_TYPE_UTIL,<BR>
" ib_thread(%d) poll_fd: hca[%d]=%p,
async=%d"<BR>
" pipe=%d cm=%d
cq=d\n",<BR>
getpid(), hca, ufds[idx-1].fd, <BR>@@ -783,14 +782,14 @@ void
dapli_thread(void *arg)
<BR>
dapl_os_unlock(&g_hca_lock);<BR>
ret = poll(ufds, fds, -1);
<BR> if (ret
<= 0)
{<BR>-
dapl_dbg_log(DAPL_DBG_TYPE_WARN,<BR>+
dapl_dbg_log(DAPL_DBG_TYPE_UTIL,<BR>
" ib_thread(%d): ERR %s
poll\n",<BR>
getpid(),strerror(errno));<BR>
dapl_os_lock(&g_hca_lock);<BR>
continue;<BR>
}<BR> <BR>-
dapl_dbg_log(DAPL_DBG_TYPE_CM,<BR>+
dapl_dbg_log(DAPL_DBG_TYPE_UTIL,<BR>
" ib_thread(%d) poll_event:
"<BR>
" async=0x%x pipe=0x%x cm=0x%x
cq=0x%x\n",<BR>
getpid(), ufds[idx-1].revents, ufds[0].revents, <BR>@@ -834,3 +833,63 @@ void
dapli_thread(void *arg) <BR>
dapl_os_unlock(&g_hca_lock); <BR> }<BR> <BR>+#ifdef
DAPL_PROVIDER_SPECIFIC_ATTR<BR>+/*<BR>+ *
dapls_set_provider_specific_attr<BR>+ *<BR>+ * Input:<BR>+ *
attr_ptr Pointer provider attributes<BR>+ *<BR>+ *
Output:<BR>+ * none<BR>+ *<BR>+ * Returns:<BR>+ * void<BR>+
*/<BR>+DAT_NAMED_ATTR ib_attrs[] = {<BR>+ <BR>+#ifdef
DAPL_EXTENSIONS<BR>+ {<BR>+
DAT_EXT_ATTR,<BR>+ DAT_EXT_ATTR_TRUE<BR>+
},<BR>+ {<BR>+
DAT_EXT_ATTR_RDMA_WRITE_IMMED,<BR>+
DAT_EXT_ATTR_TRUE<BR>+ },<BR>+
{<BR>+ DAT_EXT_ATTR_RECV_IMMED,<BR>+
DAT_EXT_ATTR_TRUE<BR>+ },<BR>+ /* inbound
immediate data placed in event, NOT payload */<BR>+
{<BR>+ DAT_EXT_ATTR_RECV_IMMED_EVENT,<BR>+
DAT_EXT_ATTR_TRUE<BR>+ },<BR>+
{<BR>+ DAT_EXT_ATTR_FETCH_AND_ADD,<BR>+
DAT_EXT_ATTR_TRUE<BR>+ },<BR>+
{<BR>+ DAT_EXT_ATTR_CMP_AND_SWAP,<BR>+
DAT_EXT_ATTR_TRUE<BR>+
},<BR>+#else<BR>+ {<BR>+
"DAT_EXTENSION_INTERFACE",<BR>+
"FALSE"<BR>+ },<BR>+#endif<BR>+};<BR>+<BR>+#define
SPEC_ATTR_SIZE(x) ( sizeof(x)/sizeof(DAT_NAMED_ATTR) )<BR>+<BR>+void
dapls_set_provider_specific_attr(<BR>+ IN
DAT_PROVIDER_ATTR *attr_ptr )<BR>+{<BR>+
attr_ptr->num_provider_specific_attr =
SPEC_ATTR_SIZE(ib_attrs);<BR>+
attr_ptr->provider_specific_attr =
ib_attrs;<BR>+}<BR>+<BR>+#endif<BR>+<BR>Index:
dapl/openib_cma/dapl_ib_mem.c<BR>===================================================================<BR>---
dapl/openib_cma/dapl_ib_mem.c (revision 4589)<BR>+++
dapl/openib_cma/dapl_ib_mem.c (working copy)<BR>@@ -25,9 +25,9
@@<BR> <BR> /**********************************************************************<BR>
* <BR>- * MODULE: dapl_det_mem.c<BR>+ * MODULE: dapl_ib_mem.c<BR> *<BR>-
* PURPOSE: Intel DET APIs: Memory windows, registration,<BR>+ * PURPOSE:
OpenIB uCMA provider Memory windows, registration,<BR>
* and protection
domain <BR> *<BR> * $Id: $<BR>@@ -72,12 +72,10 @@
dapls_convert_privileges(IN
DAT_MEM_PRIV<BR>
access |= IBV_ACCESS_LOCAL_WRITE;<BR> if
(DAT_MEM_PRIV_REMOTE_WRITE_FLAG &
privileges)<BR>
access |= IBV_ACCESS_REMOTE_WRITE;<BR>- if
(DAT_MEM_PRIV_REMOTE_READ_FLAG &
privileges)<BR>-
access |= IBV_ACCESS_REMOTE_READ;<BR>- if
(DAT_MEM_PRIV_REMOTE_READ_FLAG &
privileges)<BR>-
access |= IBV_ACCESS_REMOTE_READ;<BR>- if
(DAT_MEM_PRIV_REMOTE_READ_FLAG & privileges)<BR>+
if (DAT_MEM_PRIV_REMOTE_READ_FLAG & privileges)
{<BR> access
|=
IBV_ACCESS_REMOTE_READ;<BR>+
access |= IBV_ACCESS_REMOTE_ATOMIC;<BR>+
}<BR> <BR> return
access;<BR> }<BR>Index:
dapl/openib_cma/dapl_ib_cm.c<BR>===================================================================<BR>---
dapl/openib_cma/dapl_ib_cm.c (revision 4589)<BR>+++
dapl/openib_cma/dapl_ib_cm.c (working copy)<BR>@@ -35,7
+35,7 @@<BR> *<BR> * Description: <BR> *<BR>-
* The uDAPL openib provider - connection management<BR>+
* The OpenIB uCMA provider - uCMA connection management<BR>
*<BR>
****************************************************************************<BR>
* Source Control
System Information<BR>@@ -592,7 +592,11 @@ dapls_ib_setup_conn_listener(IN
DAPL_IA <BR> <BR> if
(rdma_bind_addr(conn->cm_id,<BR>
(struct sockaddr *)&ia_ptr->hca_ptr->hca_address))
{<BR>- dat_status
=
dapl_convert_errno(errno,"setup_listener");<BR>+
if (errno ==
-EBUSY)<BR>+
dat_status =
DAT_CONN_QUAL_IN_USE;<BR>+
else<BR>+
dat_status =
<BR>+
dapl_convert_errno(errno,"setup_listener");<BR>
goto bail;<BR> }<BR> <BR>Index:
dapl/openib_cma/dapl_ib_qp.c<BR>===================================================================<BR>---
dapl/openib_cma/dapl_ib_qp.c (revision 4589)<BR>+++
dapl/openib_cma/dapl_ib_qp.c (working copy)<BR>@@ -25,9
+25,9
@@<BR> <BR> /**********************************************************************<BR>
*<BR>- * MODULE: dapl_det_qp.c<BR>+ * MODULE: dapl_ib_qp.c<BR> *<BR>- *
PURPOSE: QP routines for access to DET Verbs<BR>+ * PURPOSE: OpenIB uCMA QP
routines <BR> *<BR> * $Id: $<BR>
**********************************************************************/<BR>Index:
dapl/openib_cma/dapl_ib_util.h<BR>===================================================================<BR>---
dapl/openib_cma/dapl_ib_util.h (revision 4589)<BR>+++
dapl/openib_cma/dapl_ib_util.h (working copy)<BR>@@ -35,7 +35,7
@@<BR> *<BR> * Description: <BR> *<BR>-
* The uDAPL openib provider - definitions, prototypes,<BR>+
* The OpenIB uCMA provider - definitions, prototypes,<BR>
*<BR>
****************************************************************************<BR>
* Source Control
System Information<BR>Index:
dapl/openib_cma/README<BR>===================================================================<BR>---
dapl/openib_cma/README (revision 4589)<BR>+++
dapl/openib_cma/README (working copy)<BR>@@ -23,15 +23,22 @@
New files for openib_scm provider<BR>
dapl/openib_cma/dapl_ib_util.c<BR>
dapl/openib_cma/dapl_ib_util.h<BR>
dapl/openib_cma/dapl_ib_cm.c<BR>+
dapl/openib_cma/dapl_ib_extensions.c<BR> <BR> A simple dapl test
just for openib_scm testing...<BR> <BR>
test/dtest/dtest.c<BR>+
test/dtest/dtest_ext.c<BR>
test/dtest/makefile<BR> <BR>
server: dtest -s <BR>
client: dtest -h hostname<BR> <BR>+or with
extensions<BR>+<BR>+ server:
dtest_ext -s <BR>+ client:
dtest_ext -h hostname<BR>+<BR> known
issues:<BR> <BR> no memory windows support
in ibverbs, dat_create_rmr fails.<BR>Index:
dapl/openib_cma/dapl_ib_cq.c<BR>===================================================================<BR>---
dapl/openib_cma/dapl_ib_cq.c (revision 4589)<BR>+++
dapl/openib_cma/dapl_ib_cq.c (working copy)<BR>@@ -35,7
+35,7 @@<BR> *<BR> * Description: <BR> *<BR>-
* The uDAPL openib provider - completion queue<BR>+ *
The OpenIB uCMA provider - completion queue<BR> *<BR>
****************************************************************************<BR>
* Source Control
System Information<BR>@@ -498,7 +498,10 @@ dapls_ib_wait_object_wait(IN
ib_wait_obj<BR> if (timeout !=
DAT_TIMEOUT_INFINITE)<BR>
timeout_ms = timeout/1000;<BR> <BR>- status =
poll(&cq_fd, 1, timeout_ms);<BR>+ /* restart
syscall */<BR>+ while ((status = poll(&cq_fd, 1,
timeout_ms)) == -1
)<BR>+ if (errno
==
EINTR)<BR>+
continue;<BR> <BR> /* returned event
*/<BR> if (status > 0) {<BR>@@ -511,6 +514,8
@@ dapls_ib_wait_object_wait(IN ib_wait_obj<BR>
/* timeout */<BR> } else if (status == 0)
<BR> status
= ETIMEDOUT;<BR>+ else
<BR>+ status =
errno;<BR> <BR>
dapl_dbg_log(DAPL_DBG_TYPE_UTIL,
<BR>
" cq_object_wait: RET evd %p ibv_cq %p ibv_ctx %p
%s\n",<BR>Index:
dat/include/dat/dat_redirection.h<BR>===================================================================<BR>---
dat/include/dat/dat_redirection.h (revision
4589)<BR>+++ dat/include/dat/dat_redirection.h
(working copy)<BR>@@ -59,10 +59,10 @@ typedef struct dat_provider
DAT_PROVIDER<BR> * This would allow a good compiler to avoid indirection
overhead when<BR> * making function calls.<BR>
*/<BR>-<BR> #define DAT_HANDLE_TO_PROVIDER(handle) (*(DAT_PROVIDER
**)(handle))<BR> #endif<BR> <BR>+<BR> #define DAT_IA_QUERY(ia,
evd, ia_msk, ia_ptr, p_msk, p_ptr) \<BR>
(*DAT_HANDLE_TO_PROVIDER (ia)->ia_query_func)
(\<BR>
(ia),
\<BR>@@ -395,6 +395,12 @@ typedef struct dat_provider
DAT_PROVIDER<BR>
(lbuf),
\<BR>
(cookie))<BR> <BR>+#define DAT_EXTENSION(handle, op, args)
\<BR>+ (*DAT_HANDLE_TO_PROVIDER
(handle)->extension_func)
(\<BR>+ (handle),
\<BR>+ (op),
\<BR>+
(args))<BR>+<BR> /***************************************************************<BR>
*<BR> * FUNCTION PROTOTYPES<BR>@@ -720,4 +726,11 @@ typedef DAT_RETURN
(*DAT_SRQ_POST_RECV_F<BR>
IN DAT_LMR_TRIPLET *, /*
local_iov
*/<BR> IN
DAT_DTO_COOKIE ); /*
user_cookie
*/<BR> <BR>+/* Extension function */<BR>+#include
<stdarg.h><BR>+typedef DAT_RETURN (*DAT_EXTENSION_FUNC)
(<BR>+ IN
DAT_HANDLE, /* dat
handle
*/<BR>+ IN
DAT_EXT_OP, /* extension operation
*/<BR>+ IN va_list
); /*
va_list */<BR>+<BR> #endif /*
_DAT_REDIRECTION_H_ */<BR>Index:
dat/include/dat/dat.h<BR>===================================================================<BR>---
dat/include/dat/dat.h (revision 4589)<BR>+++
dat/include/dat/dat.h (working copy)<BR>@@ -854,11
+854,15 @@ typedef enum dat_event_number<BR>
DAT_ASYNC_ERROR_EP_BROKEN
= 0x08003,<BR>
DAT_ASYNC_ERROR_TIMED_OUT
=
0x08004,<BR>
DAT_ASYNC_ERROR_PROVIDER_INTERNAL_ERROR =
0x08005,<BR>-
DAT_SOFTWARE_EVENT
= 0x10001<BR>+
DAT_SOFTWARE_EVENT
= 0x10001,<BR>+
DAT_EXTENSION_EVENT
= 0x20001<BR>+<BR> } DAT_EVENT_NUMBER;<BR> <BR>-/* <st1:place
w:st="on">Union</st1:place> for event Data */<BR>+/* include extension data
definitions */<BR>+#include <dat/dat_extensions.h><BR> <BR>+/*
<st1:place w:st="on">Union</st1:place> for event Data */<BR> typedef
union dat_event_data<BR> {<BR>
DAT_DTO_COMPLETION_EVENT_DATA
dto_completion_event_data;<BR>@@ -867,6 +871,7 @@ typedef union
dat_event_data<BR>
DAT_CONNECTION_EVENT_DATA
connect_event_data;<BR>
DAT_ASYNCH_ERROR_EVENT_DATA
asynch_error_event_data;<BR>
DAT_SOFTWARE_EVENT_DATA
software_event_data;<BR>+
DAT_EXTENSION_DATA
extension_data;<BR> } DAT_EVENT_DATA;<BR> <BR> /* Event struct
that holds all event information */<BR>@@ -1222,6 +1227,11 @@ extern
DAT_RETURN dat_srq_set_lw (<BR>
IN
DAT_SRQ_HANDLE, /*
srq_handle
*/<BR> IN
DAT_COUNT);
/* low_watermark
*/<BR> <BR>+extern DAT_RETURN dat_extension(<BR>+
IN DAT_HANDLE,<BR>+
IN DAT_EXT_OP,<BR>+
IN ... );<BR>+<BR> /*<BR> * DAT registry
functions.<BR> *<BR>Index:
dat/include/dat/udat_redirection.h<BR>===================================================================<BR>---
dat/include/dat/udat_redirection.h (revision 4589)<BR>+++
dat/include/dat/udat_redirection.h (working copy)<BR>@@
-199,7 +199,6 @@ typedef DAT_RETURN (*DAT_EVD_SET_UNWAITA<BR> typedef
DAT_RETURN (*DAT_EVD_CLEAR_UNWAITABLE_FUNC)
(<BR> IN
DAT_EVD_HANDLE);
/* evd_handle
*/<BR> <BR>-<BR> #include
<dat/dat_redirection.h><BR> <BR> struct dat_provider<BR>@@
-294,6 +293,10 @@ struct dat_provider<BR>
DAT_SRQ_QUERY_FUNC
srq_query_func;<BR>
DAT_SRQ_RESIZE_FUNC
srq_resize_func;<BR>
DAT_SRQ_SET_LW_FUNC
srq_set_lw_func;<BR>+<BR>+ /* extension for provder specific
functions */<BR>+
DAT_EXTENSION_FUNC
extension_func;<BR>+<BR> };<BR> <BR> #endif /*
_UDAT_REDIRECTION_H_ */<BR>Index:
dat/include/dat/dat_extensions.h<BR>===================================================================<BR>---
dat/include/dat/dat_extensions.h (revision
0)<BR>+++ dat/include/dat/dat_extensions.h
(revision 0)<BR>@@ -0,0 +1,209 @@<BR>+/*<BR>+ * Copyright (c) 2002-2005,
Network Appliance, Inc. All rights reserved.<BR>+ *<BR>+ * This Software is
licensed under one of the following licenses:<BR>+ * <BR>+ * 1) under the
terms of the "Common Public License 1.0" a copy of which is<BR>+
* in the file LICENSE.txt in the root directory. The license
is also<BR>+ * available from the Open Source Initiative,
see <BR>+ * http://www.opensource.org/licenses/cpl.php.<BR>+
* <BR>+ * 2) under the terms of the "The BSD License" a copy of which is in
the file<BR>+ * LICENSE2.txt in the root directory. The
license is also available from<BR>+ * the Open Source
Initiative, see<BR>+ *
http://www.opensource.org/licenses/bsd-license.php.<BR>+ * <BR>+ * 3) under
the terms of the "GNU General Public License (GPL) Version 2" a<BR>+
* copy of which is in the file LICENSE3.txt in the root
directory. The<BR>+ * license is also available from the
Open Source Initiative, see<BR>+ *
http://www.opensource.org/licenses/gpl-license.php.<BR>+ * <BR>+ * Licensee
has the right to choose one of the above licenses.<BR>+ * <BR>+ *
Redistributions of source code must retain the above copyright<BR>+ * notice
and one of the license notices.<BR>+ * <BR>+ * Redistributions in binary form
must reproduce both the above copyright<BR>+ * notice, one of the license
notices in the documentation<BR>+ * and/or other materials provided with the
distribution.<BR>+
*/<BR>+/**********************************************************************<BR>+
*<BR>+ * HEADER: dat_extensions.h<BR>+ *<BR>+ * PURPOSE: defines the
extensions to the DAT API for uDAPL.<BR>+ *<BR>+ * Description: Header file
for "uDAPL: User Direct Access Programming<BR>+
* Library, Version: 1.2"<BR>+
*<BR>+ * Mapping rules:<BR>+ * All global
symbols are prepended with "DAT_" or "dat_"<BR>+
* All DAT objects have an 'api' tag which, such
as 'ep' or 'lmr'<BR>+ * The method table is in
the provider definition structure.<BR>+ *<BR>+ *<BR>+
**********************************************************************/<BR>+<BR>+#ifndef
_DAT_EXTENSIONS_H_<BR>+<BR>+extern int dat_extensions;<BR>+<BR>+/* <BR>+ *
Provider specific attribute strings for extension support <BR>+ *
returned with dat_ia_query() and <BR>+ * DAT_PROVIDER_ATTR_MASK ==
DAT_PROVIDER_FIELD_PROVIDER_SPECIFIC_ATTR<BR>+ *<BR>+ *
DAT_NAMED_ATTR name == extended operation, <BR>+
*
value == TRUE if extended operation is supported<BR>+ */<BR>+#define
DAT_EXT_ATTR
"DAT_EXTENSION_INTERFACE"<BR>+#define
DAT_EXT_ATTR_RDMA_WRITE_IMMED
"DAT_EXT_RDMA_WRITE_IMMED"<BR>+#define
DAT_EXT_ATTR_RECV_IMMED
"DAT_EXT_RECV_IMMED"<BR>+#define
DAT_EXT_ATTR_RECV_IMMED_EVENT
"DAT_EXT_RECV_IMMED_EVENT"<BR>+#define DAT_EXT_ATTR_RECV_IMMED_PAYLOAD
"DAT_EXT_RECV_IMMED_PAYLOAD"<BR>+#define DAT_EXT_ATTR_FETCH_AND_ADD
"DAT_EXT_FETCH_AND_ADD"<BR>+#define DAT_EXT_ATTR_CMP_AND_SWAP
"DAT_EXT_CMP_AND_SWAP"<BR>+#define
DAT_EXT_ATTR_TRUE
"TRUE"<BR>+#define
DAT_EXT_ATTR_FALSE
"FALSE"<BR>+<BR>+/* <BR>+ * Extension OPERATIONS <BR>+ */<BR>+typedef enum
dat_ext_op<BR>+{<BR>+
DAT_EXT_RDMA_WRITE_IMMED,<BR>+
DAT_EXT_RECV_IMMED,<BR>+
DAT_EXT_FETCH_AND_ADD,<BR>+
DAT_EXT_CMP_AND_SWAP,<BR>+<BR>+} DAT_EXT_OP;<BR>+<BR>+/* <BR>+ * Extension
completion event TYPES<BR>+ */<BR>+typedef enum
dat_ext_event_type<BR>+{<BR>+
DAT_EXT_RDMA_WRITE_IMMED_STATUS = 1,<BR>+
DAT_EXT_RECV_NO_IMMED,<BR>+
DAT_EXT_RECV_IMMED_DATA_EVENT,
<BR>+
DAT_EXT_RECV_IMMED_DATA_PAYLOAD,
<BR>+
DAT_EXT_FETCH_AND_ADD_STATUS,
<BR>+
DAT_EXT_CMP_AND_SWAP_STATUS,
<BR>+<BR>+} DAT_EXT_EVENT_TYPE;<BR>+<BR>+/* <BR>+ * Extension completion event
DATA<BR>+ */<BR>+typedef struct dat_immediate_data
<BR>+{<BR>+ DAT_UINT32 data;<BR>+<BR>+}
DAT_RDMA_WRITE_IMMED_DATA;<BR>+<BR>+typedef struct
dat_extension_data<BR>+{<BR>+
DAT_DTO_COMPLETION_EVENT_DATA dto;<BR>+
DAT_EXT_EVENT_TYPE
type;<BR>+ union
{<BR>+
DAT_RDMA_WRITE_IMMED_DATA
immed; <BR>+ }
val;<BR>+} DAT_EXTENSION_DATA;<BR>+<BR>+typedef enum
dat_ext_flags<BR>+{<BR>+
DAT_EXT_WRITE_IMMED_FLAG =
0x1,<BR>+ DAT_EXT_WRITE_CONFIRM_FLAG
= 0x2,<BR>+<BR>+} DAT_EXT_FLAGS;<BR>+<BR>+/* <BR>+ * Extended API with
redirection via DAT extension function<BR>+ */<BR>+<BR>+/* <BR>+ * RDMA Write
with IMMEDIATE extension:<BR>+ *<BR>+ * Asynchronous call performs a normal
RDMA write to the remote endpoint <BR>+ * followed by a post of an extended
immediate data value to the receive <BR>+ * EVD on the remote endpoint. Event
completion for the request completes <BR>+ * as an DAT_EXTENSION_EVENT with
type set to DAT_EXT_RDMA_WRITE_IMMED_STATUS.<BR>+ * Event completion on the
remote endpoint completes as an DAT_EXTENSION_EVENT<BR>+ * with type set to
DAT_EXT_RECV_IMMED_DATA_IN_EVENT or <BR>+ * DAT_EXT_RECV_IMMED_DATA_IN_PAYLOAD
depending on the provider transport.<BR>+ *<BR>+ * DAT_EXT_WRITE_IMMED_FLAG
requests that the supplied<BR>+ *'immediate' value be sent as the payload of a
four byte send following<BR>+ * the RDMA Write, or any transport-dependent
equivalent thereof.<BR>+ * For example, on InfiniBand the request should be
translated as an<BR>+ * RDMA Write with Immediate. <BR>+ *<BR>+ *
DAT_EXT_WRITE_CONFIRM_FLAG requests that this DTO<BR>+ * not complete until
receipt by the far end is confirmed. <BR>+ *<BR>+ * Note to Consumers: the
immediate data will consume a receive<BR>+ * buffer at the Data Sink. <BR>+
*<BR>+ * Other extension flags:<BR>+ * n/a<BR>+ */<BR>+#define
dat_ep_post_rdma_write_immed(ep, size, lbuf, cookie, rbuf, idata, eflgs, flgs)
\<BR>+
dat_extension( ep,
\<BR>+
DAT_EXT_RDMA_WRITE_IMMED,
\<BR>+
(size),
\<BR>+
(lbuf),
\<BR>+
(cookie),
\<BR>+
(rbuf),
\<BR>+
(idata),
\<BR>+
(eflgs),
\<BR>+
(flgs))<BR>+ <BR>+/* <BR>+ * Call performs a normal
post receive message to the local endpoint<BR>+ * that includes additional
32-bit buffer space for immediate data <BR>+ * Event completion for the
request completes as an <BR>+ * DAT_EXTENSION_EVENT with type set to
DAT_EXT_RDMA_WRITE_IMMED_STATUS.<BR>+ */<BR>+#define
dat_ep_post_recv_immed(ep, size, lbuf, cookie, flgs)
\<BR>+
dat_extension( ep,
\<BR>+
DAT_EXT_RECV_IMMED,
\<BR>+
(size),
\<BR>+
(lbuf),
\<BR>+
(cookie),
\<BR>+
(flgs))<BR>+<BR>+/*<BR>+ * This asynchronous call is modeled after the
InfiniBand atomic <BR>+ * Fetch and Add operation. The add_value is added to
the 64 bit <BR>+ * value stored at the remote memory location specified in
remote_iov<BR>+ * and the result is stored in the local_iov. <BR>+
*/<BR>+#define dat_ep_post_fetch_and_add(ep, add_val, lbuf, cookie, rbuf,
flgs) \<BR>+
dat_extension( ep,
\<BR>+
DAT_EXT_FETCH_AND_ADD,
\<BR>+
(add_val),
\<BR>+
(lbuf),
\<BR>+
(cookie),
\<BR>+
(rbuf),
\<BR>+
(flgs))<BR>+
<BR>+/*<BR>+ * This asynchronous call is modeled after the InfiniBand atomic
<BR>+ * Compare and Swap operation. The cmp_value is compared to the 64 bit
<BR>+ * value stored at the remote memory location specified in
remote_iov. <BR>+ * If the two values are equal, the 64 bit swap_value
is stored in <BR>+ * the remote memory location. In all cases, the
original 64 bit <BR>+ * value stored in the remote memory location is copied
to the local_iov.<BR>+ */<BR>+#define dat_ep_post_cmp_and_swap(ep, cmp_val,
swap_val, lbuf, cookie, rbuf, flgs)
\<BR>+
dat_extension( ep,
\<BR>+
DAT_EXT_CMP_AND_SWAP,
\<BR>+
(cmp_val),
\<BR>+
(swap_val),
\<BR>+
(lbuf),
\<BR>+
(cookie),
\<BR>+
(rbuf),
\<BR>+
(flgs))<BR>+<BR>+#endif /* _DAT_EXTENSIONS_H_ */<BR>+<BR>Index:
dat/common/dat_api.c<BR>===================================================================<BR>---
dat/common/dat_api.c (revision 4594)<BR>+++
dat/common/dat_api.c (working copy)<BR>@@
-1142,6 +1142,36 @@ DAT_RETURN
dat_srq_set_lw(<BR>
low_watermark);<BR> }<BR> <BR>+DAT_RETURN
dat_extension(<BR>+
IN DAT_HANDLE
handle,<BR>+
IN DAT_EXT_OP
ext_op,<BR>+
IN ... )<BR>+<BR>+{<BR>+
DAT_RETURN status;<BR>+ va_list
args;<BR>+<BR>+ if (handle ==
NULL)<BR>+
{<BR>+ return
DAT_ERROR(DAT_INVALID_HANDLE,
DAT_INVALID_HANDLE_EP);<BR>+
}<BR>+<BR>+ /* verify provider extension support
*/<BR>+ if
(!dat_extensions)<BR>+
{<BR>+ return
DAT_ERROR(DAT_NOT_IMPLEMENTED, 0);<BR>+
}<BR>+<BR>+ va_start(args,
ext_op);<BR>+<BR>+ status =
DAT_EXTENSION(handle,<BR>+
ext_op,<BR>+
args);<BR>+
va_end(args);<BR>+<BR>+ return
status;<BR>+}<BR>+<BR> /*<BR> * Local variables:<BR> *
c-indent-level: 4<BR>Index:
dat/udat/udat.c<BR>===================================================================<BR>---
dat/udat/udat.c (revision 4594)<BR>+++
dat/udat/udat.c (working copy)<BR>@@ -66,6 +66,10 @@
udat_check_state ( void );<BR>
*
*<BR>
*********************************************************************/<BR> <BR>+/*<BR>+
* Use a global to get an unresolved when run with pre-extension
library<BR>+ */<BR>+int dat_extensions = 0;<BR> <BR> /*<BR>
*<BR>@@ -230,13 +234,44 @@ dat_ia_openv
(<BR>
async_event_qlen,<BR>
async_event_handle,<BR>
ia_handle);<BR>+<BR>+ /*<BR>+
* See if provider supports extensions<BR>+
*/<BR> if (dat_status ==
DAT_SUCCESS)<BR> {<BR>-
return_handle = dats_set_ia_handle (*ia_handle);<BR>-
if (return_handle >= 0)<BR>-
{<BR>- *ia_handle =
(DAT_IA_HANDLE)return_handle;<BR>-
}<BR>+
DAT_PROVIDER_ATTR
p_attr;<BR>+
int
i;<BR>+<BR>+ return_handle =
dats_set_ia_handle
(*ia_handle);<BR>+ if (return_handle
>= 0)<BR>+
{<BR>+
*ia_handle =
(DAT_IA_HANDLE)return_handle;<BR>+
}<BR>+<BR>+ if ( dat_ia_query(
*ia_handle,<BR>+
NULL,<BR>+
0,<BR>+
NULL,<BR>+
DAT_PROVIDER_FIELD_PROVIDER_SPECIFIC_ATTR,<BR>+
&p_attr ) == DAT_SUCCESS )<BR>+
{<BR>+ for (
i = 0; i < p_attr.num_provider_specific_attr; i++
)<BR>+
{<BR>+
if ( (strcmp(
p_attr.provider_specific_attr[i].name,<BR>+
"DAT_EXTENSION_INTERFACE"
) == 0)
&&<BR>+
(strcmp(
p_attr.provider_specific_attr[i].value,<BR>+
"TRUE" ) == 0)
)<BR>+
{<BR>+
dat_os_dbg_print(DAT_OS_DBG_TYPE_CONSUMER_API,<BR>+
"DAT Registry: dat_ia_open ()
"<BR>+
"DAPL Extension Interface
supported!\n");<BR>+<BR>+
dat_extensions =
1;<BR>+
break;<BR>+
}<BR>+
}<BR>+ }<BR>
}<BR> <BR> return dat_status;<BR>Index:
README<BR>===================================================================<BR>---
README (revision 4589)<BR>+++ README (working copy)<BR>@@ -1,5
+1,10 @@<BR> There are now 3 uDAPL providers for openib
(openib,openib_scm,openib_cma). <BR> <BR>+NEW FEATURES for openib_cma
provider:<BR>+API extensions for immediate data and atomic operations have
been added.<BR>+see dat/include/dat/dat_extensions.h for new API's. <BR>+see
dapl/test/dtest/dtest_ext.c for example test cast
<BR>+<BR> <BR> ==========<BR> 1.0
BUILD:<BR><o:p> </o:p></SPAN></FONT></P><PRE><FONT face="Courier New" size=2><SPAN style="FONT-SIZE: 10pt"><o:p> </o:p></SPAN></FONT></PRE>
<P class=MsoNormal><FONT face=Arial size=2><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: Arial"><o:p> </o:p></SPAN></FONT></P></DIV></BLOCKQUOTE></BODY></HTML>