<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML xmlns="http://www.w3.org/TR/REC-html40" xmlns:v =
"urn:schemas-microsoft-com:vml" xmlns:o =
"urn:schemas-microsoft-com:office:office" xmlns:w =
"urn:schemas-microsoft-com:office:word" xmlns:st1 =
"urn:schemas-microsoft-com:office:smarttags"><HEAD>
<META http-equiv=Content-Type content="text/html; charset=us-ascii">
<META content="MSHTML 6.00.2900.2769" name=GENERATOR><o:SmartTagType
namespaceuri="urn:schemas-microsoft-com:office:smarttags" name="place"
downloadurl="http://www.5iantlavalamp.com/"></o:SmartTagType><!--[if !mso]>
<STYLE>st1\:* {
BEHAVIOR: url(#default#ieooui)
}
</STYLE>
<![endif]-->
<STYLE>@font-face {
font-family: Courier;
}
@page Section1 {size: 8.5in 11.0in; margin: 1.0in 1.25in 1.0in 1.25in; }
P.MsoNormal {
FONT-SIZE: 11pt; MARGIN: 3pt 0in; FONT-FAMILY: "Times New Roman"
}
LI.MsoNormal {
FONT-SIZE: 11pt; MARGIN: 3pt 0in; FONT-FAMILY: "Times New Roman"
}
DIV.MsoNormal {
FONT-SIZE: 11pt; MARGIN: 3pt 0in; FONT-FAMILY: "Times New Roman"
}
A:link {
COLOR: blue; TEXT-DECORATION: underline
}
SPAN.MsoHyperlink {
COLOR: blue; TEXT-DECORATION: underline
}
A:visited {
COLOR: purple; TEXT-DECORATION: underline
}
SPAN.MsoHyperlinkFollowed {
COLOR: purple; TEXT-DECORATION: underline
}
P.MsoPlainText {
FONT-SIZE: 10pt; MARGIN: 0in 0in 0pt; FONT-FAMILY: "Courier New"
}
LI.MsoPlainText {
FONT-SIZE: 10pt; MARGIN: 0in 0in 0pt; FONT-FAMILY: "Courier New"
}
DIV.MsoPlainText {
FONT-SIZE: 10pt; MARGIN: 0in 0in 0pt; FONT-FAMILY: "Courier New"
}
PRE {
FONT-SIZE: 10pt; MARGIN: 0in 0in 0pt; FONT-FAMILY: "Courier New"
}
SPAN.EmailStyle17 {
COLOR: windowtext; FONT-FAMILY: Arial; mso-style-type: personal-compose
}
DIV.Section1 {
page: Section1
}
</STYLE>
<!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></HEAD>
<BODY lang=EN-US vLink=purple link=blue>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=635100720-23122005>Actually,</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=635100720-23122005>iWARP does not support neither immediate data nor
atomic ops.</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=635100720-23122005>Arkady</SPAN></FONT></DIV>
<DIV> </DIV><o:SmartTagType
namespaceuri="urn:schemas-microsoft-com:office:smarttags"
name="Street"></o:SmartTagType><o:SmartTagType
namespaceuri="urn:schemas-microsoft-com:office:smarttags"
name="address"></o:SmartTagType>
<STYLE>st1\:* {
BEHAVIOR: url(#ieooui)
}
</STYLE>
<STYLE>@page Section1 {size: 8.5in 11.0in; margin: 1.0in 1.25in 1.0in 1.25in; mso-header-margin: .5in; mso-footer-margin: .5in; mso-paper-source: 0; }
P.MsoNormal {
FONT-SIZE: 12pt; MARGIN: 0in 0in 0pt; FONT-FAMILY: "Times New Roman"; mso-style-parent: ""; mso-pagination: widow-orphan; mso-fareast-font-family: "Times New Roman"
}
LI.MsoNormal {
FONT-SIZE: 12pt; MARGIN: 0in 0in 0pt; FONT-FAMILY: "Times New Roman"; mso-style-parent: ""; mso-pagination: widow-orphan; mso-fareast-font-family: "Times New Roman"
}
DIV.MsoNormal {
FONT-SIZE: 12pt; MARGIN: 0in 0in 0pt; FONT-FAMILY: "Times New Roman"; mso-style-parent: ""; mso-pagination: widow-orphan; mso-fareast-font-family: "Times New Roman"
}
A:link {
COLOR: blue; TEXT-DECORATION: underline; text-underline: single
}
SPAN.MsoHyperlink {
COLOR: blue; TEXT-DECORATION: underline; text-underline: single
}
A:visited {
COLOR: purple; TEXT-DECORATION: underline; text-underline: single
}
SPAN.MsoHyperlinkFollowed {
COLOR: purple; TEXT-DECORATION: underline; text-underline: single
}
SPAN.SpellE {
mso-style-name: ""; mso-spl-e: yes
}
SPAN.GramE {
mso-style-name: ""; mso-gram-e: yes
}
DIV.Section1 {
page: Section1
}
</STYLE>
<DIV class=Section1>
<P class=MsoNormal align=left><SPAN class=SpellE><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Arkady</SPAN></SPAN><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: Arial"> <SPAN
class=SpellE>Kanevsky</SPAN><SPAN
style="mso-tab-count: 2">
</SPAN>email: <A
href="mailto:arkady@netapp.com">arkady@netapp.com</A><o:p></o:p></SPAN></P>
<P class=MsoNormal><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Network
Appliance Inc.<SPAN
style="mso-tab-count: 2">
</SPAN><SPAN class=GramE>phone</SPAN>: 781-768-5395<o:p></o:p></SPAN></P>
<P class=MsoNormal><st1:Street w:st="on"><st1:address
style="BACKGROUND-POSITION: left bottom; BACKGROUND-IMAGE: url(res://ietag.dll/#34/#1001); BACKGROUND-REPEAT: repeat-x"
tabIndex=0 w:st="on"><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">1601
Trapelo Rd. - Suite 16.</SPAN></st1:address></st1:Street><SPAN
style="mso-tab-count: 2"> </SPAN><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Fax:
781-895-1195<o:p></o:p></SPAN></P>
<P class=MsoNormal><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Waltham, MA
02451
central phone: 781-768-5300</SPAN></P></DIV>
<DIV> </DIV><BR>
<BLOCKQUOTE
style="PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT: #0000ff 2px solid; MARGIN-RIGHT: 0px">
<DIV class=OutlookMessageHeader lang=en-us dir=ltr align=left>
<HR tabIndex=-1>
<FONT face=Tahoma size=2><B>From:</B> Kanevsky, Arkady <BR><B>Sent:</B>
Friday, December 23, 2005 2:55 PM<BR><B>To:</B> Arlin Davis; Lentini,
James<BR><B>Cc:</B> openib-general@openib.org;
dat-discussions@yahoogroups.com<BR><B>Subject:</B> [dat-discussions] RE:
[RFC][PATCH] OpenIB uDAPL extension proposal - sample immed data and atomic
api's<BR></FONT><BR></DIV>
<DIV></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>Arlin,</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>nice proposal, thanks.</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>I have one high level question and a few specific
technical ones.</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005></SPAN></FONT> </DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>1. Why do you want to provide this functionality via
extension instead of part of new DAT spec, say 2.0?</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>This will allow Consumers to use all events,
operations, and Provider/IA functionality uniformly
instead</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>of via 2 separate layers. This will also ensure that
this basic funcionality can be provided by all DAPL
Provider</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>the same way on DAPL and DAT
layers.</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>DAPL 2.0 is not done yet so we have time to
incorporate that.</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>DAPL 2.0 already introduced new functionality which
is easy to beef up for your proposal.</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT size=2><SPAN class=554400019-23122005><FONT
face=Arial><FONT color=#0000ff>See <FONT size=2>DAT_DTOS for example.
DAT_EVENT is also modified to handle remote invalidation
so</FONT></FONT></FONT></SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>a small addition for Immediate data and Atoimc ops is
a sensible addition.</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>This should simplify proposal significantly. As you
will not need to introduce any new</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>EXT structures.</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005></SPAN></FONT> </DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>In general, extension route was intended for RNIC|HCA
providers to expose HW capabilities beyond</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>IBTA, iWARP and VIA standards. The standard RDMA
functionality is best handle via spec addition.</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>DAT 2.0 does it for FMR, remote and local memory
invalidation as well as others.</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005></SPAN></FONT> </DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>I had posted a complete list of changes/addition to
DAT 2.0 about a month ago.</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>But we had not discussed yet version change from 1.3
to 2.0 nor how much backwards compatibility spec</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>will provide.</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005></SPAN></FONT> </DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>2. What is IMMED_EVENT? is it just immediate data
without any payload one?</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>I suggest chnaging the name so it will not use
"EVENT". Just call it NO_PAYLOAD.</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>Do you want to support 2 different way to delivery
immediate data?</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>One in event and one in data
payload?</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>Why? I would think that just an event way will
do.</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005></SPAN></FONT> </DIV>
<DIV dir=ltr align=left><FONT size=2><SPAN class=554400019-23122005><FONT
color=#0000ff><FONT face=Arial>3. I suggest beefing up <FONT
size=2>DAT_DTO_COMPLETION_EVENT_DATA and DAT_DTOS
</FONT></FONT></FONT></SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>to convey which operation completed and return
Immediate data if complete operation had immediate data.</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>Since we already modified these 2 struct as part of
DAT 2.0 change lets add your proposal to the change.</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>This will allow Consumers to use single approach to
deal with completions, extension to the current one</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>but not a structural one. No need for
DAT_EXTENSION_DATA, DAT_EXT_EVENT_TYPE, DAT_EXT_OP</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>nor the whole mechanism for extended
ops.</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005></SPAN></FONT> </DIV>
<DIV dir=ltr align=left><FONT face=Arial><FONT size=2><FONT
color=#0000ff><FONT size=+0><SPAN class=554400019-23122005>4. What is the
purpose of DAT_EXT_WRITE_CONFIRM_FLAG? Is it to expose IB round trip
semantic?</SPAN></FONT></FONT></FONT></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial><FONT size=2><FONT
color=#0000ff><FONT size=+0><SPAN class=554400019-23122005>iWARP does not
support immediate data. One can try to format payload to pass immediate
data.</SPAN></FONT></FONT></FONT></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial><FONT size=2><FONT
color=#0000ff><FONT size=+0><SPAN class=554400019-23122005>Is that what you
had in mind?</SPAN></FONT></FONT></FONT></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial><FONT size=2><FONT
color=#0000ff><FONT size=+0><SPAN
class=554400019-23122005></SPAN></FONT></FONT></FONT></FONT> </DIV>
<DIV dir=ltr align=left><FONT face=Arial><FONT size=2><FONT
color=#0000ff><FONT size=+0><SPAN class=554400019-23122005>What is the
sema</SPAN></FONT><FONT size=+0><SPAN class=554400019-23122005>ntic meaning of
the completion with this flag set? without flag
set?</SPAN></FONT></FONT></FONT></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>Are extended flags are additonal values for
COMPLETION_FLAGS? 2.4.1 talks about extended flags </SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>but where they are passed in is not
defined.</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>DAT 2.0 extended them already for FMR barrier. I
would prefer to follow that route rather than creating a
separate</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>extension completion flags.</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005></SPAN></FONT> </DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>5. Why do you need RECV_IMMED? If Immed data is
delivered in event no new Recv operation is needed.</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>If Consumer asks for immediate data in payload where
in payload will it be?</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>If this is needed for local match for remote
RDMA_Write to handle immediate data lets state so.</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005></SPAN></FONT> </DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>What happens for mismatch between local and remote
op? That is recv was posted for Send and RDMA_Write</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>"arrived"? Vice Versa?</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005></SPAN></FONT><FONT face=Arial color=#0000ff
size=2><SPAN class=554400019-23122005></SPAN></FONT> </DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>6. I see extension for immediate data for rdma_write
but not for send. Is this deliberate? If we are going</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>to extend DAT semantic to support Immediate data we
can as well support the full IBTA/iWARP functionality for
it.</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005></SPAN></FONT> </DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>7. Currently memory registration do not support
access to LMR or RMR by Atomic ops.</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>Do you propose to extend the meaning of current
MEM_PRIV for LMR and RMR to covers atomic accesses</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>or add new values to LMR_MEM_PRIV and RMR_MEM_PRIV
for atomic operation support?</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005></SPAN></FONT> </DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>8. Any alignment requirements for memory used for
atomic ops?</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005></SPAN></FONT> </DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>9. Any correlation requirements for SRQ buffers to
support recv with immediate data?</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005></SPAN></FONT> </DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>Have a great holidays,</SPAN></FONT></DIV>
<DIV dir=ltr align=left><FONT face=Arial color=#0000ff size=2><SPAN
class=554400019-23122005>Arkady</SPAN></FONT></DIV>
<DIV> </DIV><o:SmartTagType
namespaceuri="urn:schemas-microsoft-com:office:smarttags"
name="Street"></o:SmartTagType><o:SmartTagType
namespaceuri="urn:schemas-microsoft-com:office:smarttags"
name="address"></o:SmartTagType>
<STYLE>st1\:* {
BEHAVIOR: url(#ieooui)
}
</STYLE>
<STYLE>@page Section1 {size: 8.5in 11.0in; margin: 1.0in 1.25in 1.0in 1.25in; mso-header-margin: .5in; mso-footer-margin: .5in; mso-paper-source: 0; }
P.MsoNormal {
FONT-SIZE: 12pt; MARGIN: 0in 0in 0pt; FONT-FAMILY: "Times New Roman"; mso-style-parent: ""; mso-pagination: widow-orphan; mso-fareast-font-family: "Times New Roman"
}
LI.MsoNormal {
FONT-SIZE: 12pt; MARGIN: 0in 0in 0pt; FONT-FAMILY: "Times New Roman"; mso-style-parent: ""; mso-pagination: widow-orphan; mso-fareast-font-family: "Times New Roman"
}
DIV.MsoNormal {
FONT-SIZE: 12pt; MARGIN: 0in 0in 0pt; FONT-FAMILY: "Times New Roman"; mso-style-parent: ""; mso-pagination: widow-orphan; mso-fareast-font-family: "Times New Roman"
}
A:link {
COLOR: blue; TEXT-DECORATION: underline; text-underline: single
}
SPAN.MsoHyperlink {
COLOR: blue; TEXT-DECORATION: underline; text-underline: single
}
A:visited {
COLOR: purple; TEXT-DECORATION: underline; text-underline: single
}
SPAN.MsoHyperlinkFollowed {
COLOR: purple; TEXT-DECORATION: underline; text-underline: single
}
SPAN.SpellE {
mso-style-name: ""; mso-spl-e: yes
}
SPAN.GramE {
mso-style-name: ""; mso-gram-e: yes
}
DIV.Section1 {
page: Section1
}
</STYLE>
<DIV class=Section1>
<P class=MsoNormal align=left><SPAN class=SpellE><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Arkady</SPAN></SPAN><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: Arial"> <SPAN
class=SpellE>Kanevsky</SPAN><SPAN
style="mso-tab-count: 2">
</SPAN>email: <A
href="mailto:arkady@netapp.com">arkady@netapp.com</A><o:p></o:p></SPAN></P>
<P class=MsoNormal><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Network
Appliance Inc.<SPAN
style="mso-tab-count: 2">
</SPAN><SPAN class=GramE>phone</SPAN>: 781-768-5395<o:p></o:p></SPAN></P>
<P class=MsoNormal><st1:Street w:st="on"><st1:address
style="BACKGROUND-POSITION: left bottom; BACKGROUND-IMAGE: url(res://ietag.dll/#34/#1001); BACKGROUND-REPEAT: repeat-x"
tabIndex=0 w:st="on"><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">1601
Trapelo Rd. - Suite 16.</SPAN></st1:address></st1:Street><SPAN
style="mso-tab-count: 2"> </SPAN><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Fax:
781-895-1195<o:p></o:p></SPAN></P>
<P class=MsoNormal><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Waltham,
MA
02451
central phone: 781-768-5300</SPAN></P></DIV>
<DIV> </DIV><BR>
<BLOCKQUOTE dir=ltr
style="PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT: #0000ff 2px solid; MARGIN-RIGHT: 0px">
<DIV class=OutlookMessageHeader lang=en-us dir=ltr align=left>
<HR tabIndex=-1>
<FONT face=Tahoma size=2><B>From:</B> Arlin Davis
[mailto:arlin.r.davis@intel.com] <BR><B>Sent:</B> Thursday, December 22,
2005 6:20 PM<BR><B>To:</B> Lentini, James; Kanevsky, Arkady<BR><B>Cc:</B>
openib-general@openib.org;
dat-discussions@yahoogroups.com<BR><B>Subject:</B> [RFC][PATCH] OpenIB uDAPL
extension proposal - sample immed data and atomic api's<BR></FONT><BR></DIV>
<DIV></DIV>
<DIV class=Section1>
<P class=MsoNormal><FONT face="Times New Roman" size=2><SPAN
style="FONT-SIZE: 11pt">James and Arkady,<o:p></o:p></SPAN></FONT></P>
<P class=MsoNormal><FONT face="Times New Roman"
size=2><o:p> </o:p></FONT></P>
<P class=MsoNormal><FONT face=Arial size=2><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">DAPL provides a generalized
abstraction to RDMA capable transports. As a generalized abstraction, it
cannot exploit the unique properties that many of the underlying
platforms/interconnects can provide so I would like to propose a simple
(minimum impact on libdat) extensible interface to uDAPL that will allow
vendors to expose such capabilities. I am looking for feedback, especially
from the DAT collaborative. I have included both a design document and
actual working code as a reference.<o:p></o:p></SPAN></FONT></P>
<P class=MsoNormal><FONT face=Arial size=2><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: Arial"><o:p> </o:p></SPAN></FONT></P>
<P class=MsoNormal><FONT face=Arial size=2><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">The patch provides a fully
tested DAT and DAPL library (openib_cma) set with the following provider
extensions:<o:p></o:p></SPAN></FONT></P>
<P class=MsoNormal><FONT face=Arial size=2><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: Arial"><o:p> </o:p></SPAN></FONT></P>
<P class=MsoNormal><FONT face=Courier size=2><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: Courier">DAT_RETURN<BR>dat_ep_post_write_immed(<BR>
IN DAT_EP_HANDLE
ep_handle,<o:p></o:p></SPAN></FONT></P>
<P class=MsoNormal
style="MARGIN-BOTTOM: 0pt; MARGIN-LEFT: 0.5in; MARGIN-RIGHT: 0in; mso-margin-top-alt: 0in"><FONT
face=Courier size=2><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Courier">IN
DAT_COUNT
num_segments<BR>IN DAT_LMR_TRIPLET
*local_iov,<BR>IN DAT_DTO_COOKIE
user_cookie,<o:p></o:p></SPAN></FONT></P>
<P class=MsoNormal
style="MARGIN-BOTTOM: 0pt; MARGIN-LEFT: 0.5in; MARGIN-RIGHT: 0in; mso-margin-top-alt: 0in"><FONT
face=Courier size=2><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Courier">IN
DAT_RMR_TRIPLE *remote_iov,<BR>IN
DAT_UINT32
immediate_data,<BR>IN DAT_COMPLETION_FLAGS
completion_flags);<o:p></o:p></SPAN></FONT></P>
<P class=MsoNormal><FONT face=Arial size=2><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: Arial"><o:p> </o:p></SPAN></FONT></P>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt"><FONT face=Courier
size=2><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: Courier">DAT_RETURN<BR>dat_ep_post_cmp_and_swap(<BR>
IN DAT_EP_HANDLE
ep_handle,<o:p></o:p></SPAN></FONT></P>
<P class=MsoNormal
style="MARGIN-BOTTOM: 0pt; MARGIN-LEFT: 0.5in; MARGIN-RIGHT: 0in; mso-margin-top-alt: 0in"><FONT
face=Courier size=2><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Courier">IN
DAT_UINT64
cmp_value,<BR>IN
DAT_UINT64
swap_value,<o:p></o:p></SPAN></FONT></P>
<P class=MsoNormal
style="MARGIN-BOTTOM: 0pt; MARGIN-LEFT: 0.5in; MARGIN-RIGHT: 0in; mso-margin-top-alt: 0in"><FONT
face=Courier size=2><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Courier">IN
DAT_LMR_TRIPLE *local_iov,<BR>IN
DAT_DTO_COOKIE
user_cookie,<o:p></o:p></SPAN></FONT></P>
<P class=MsoNormal
style="MARGIN-BOTTOM: 0pt; MARGIN-LEFT: 0.5in; MARGIN-RIGHT: 0in; mso-margin-top-alt: 0in"><FONT
face=Courier size=2><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Courier">IN
DAT_RMR_TRIPLE *remote_iov,<BR>IN
DAT_COMPLETION_FLAGS completion_flags);<o:p></o:p></SPAN></FONT></P>
<P class=MsoNormal><FONT face=Arial size=2><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: Arial"><o:p> </o:p></SPAN></FONT></P>
<P class=MsoNormal style="MARGIN: 0in 0in 0pt"><FONT face=Courier
size=2><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: Courier">DAT_RETURN<BR>dat_ep_post_fetch_and_add(<BR>
IN DAT_EP_HANDLE
ep_handle,<o:p></o:p></SPAN></FONT></P>
<P class=MsoNormal
style="MARGIN-BOTTOM: 0pt; MARGIN-LEFT: 0.5in; MARGIN-RIGHT: 0in; mso-margin-top-alt: 0in"><FONT
face=Courier size=2><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Courier">IN
DAT_UINT64
add_value,<o:p></o:p></SPAN></FONT></P>
<P class=MsoNormal
style="MARGIN-BOTTOM: 0pt; MARGIN-LEFT: 0.5in; MARGIN-RIGHT: 0in; mso-margin-top-alt: 0in"><FONT
face=Courier size=2><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Courier">IN
DAT_LMR_TRIPLE *local_iov,<BR>IN
DAT_DTO_COOKIE
user_cookie,<o:p></o:p></SPAN></FONT></P>
<P class=MsoNormal
style="MARGIN-BOTTOM: 0pt; MARGIN-LEFT: 0.5in; MARGIN-RIGHT: 0in; mso-margin-top-alt: 0in"><FONT
face=Courier size=2><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Courier">IN
DAT_RMR_TRIPLE *remote_iov,<BR>IN
DAT_COMPLETION_FLAGS completion_flags);<o:p></o:p></SPAN></FONT></P>
<P class=MsoNormal><FONT face=Arial size=2><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: Arial"><o:p> </o:p></SPAN></FONT></P>
<P class=MsoNormal><FONT face=Arial size=2><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Also, included is a sample
program (dtest_ext.c) that can be used as a programming
example.<o:p></o:p></SPAN></FONT></P>
<P class=MsoNormal><FONT face=Arial size=2><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: Arial"><o:p> </o:p></SPAN></FONT></P>
<P class=MsoNormal><FONT face=Arial size=2><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Thanks,<o:p></o:p></SPAN></FONT></P>
<P class=MsoNormal><FONT face=Arial size=2><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: Arial"><o:p> </o:p></SPAN></FONT></P>
<P class=MsoNormal><FONT face=Arial size=2><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">-arlin<o:p></o:p></SPAN></FONT></P>
<P class=MsoNormal><FONT face=Arial size=2><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: Arial"><o:p> </o:p></SPAN></FONT></P><PRE><FONT face="Courier New" size=2><SPAN style="FONT-SIZE: 10pt">Signed-off by: Arlin Davis <A href="mailto:ardavis@ichips.intel.com"><ardavis@ichips.intel.com></A><o:p></o:p></SPAN></FONT></PRE><PRE><FONT face="Courier New" size=2><SPAN style="FONT-SIZE: 10pt"><o:p> </o:p></SPAN></FONT></PRE><PRE><FONT face="Courier New" size=2><SPAN style="FONT-SIZE: 10pt"><o:p> </o:p></SPAN></FONT></PRE><PRE><FONT face="Courier New" size=2><SPAN style="FONT-SIZE: 10pt"><o:p> </o:p></SPAN></FONT></PRE>
<P class=MsoPlainText><FONT face="Courier New" size=2><SPAN
style="FONT-SIZE: 10pt">Index:
test/dtest/dat.conf<BR>===================================================================<BR>---
test/dtest/dat.conf (revision 4589)<BR>+++ test/dtest/dat.conf (working
copy)<BR>@@ -1,11 +1,20 @@<BR> #<BR>-# DAT 1.1 and 1.2 configuration
file<BR>+# DAT 1.2 configuration file<BR> #<BR> # Each entry
should have the following fields:<BR> #<BR> # <ia_name>
<api_version> <threadsafety> <default> <lib_path>
\<BR> #
<provider_version> <ia_params>
<platform_params><BR> #<BR>-# Example for openib using the first
Mellanox adapter, port 1 and port 2<BR>+# Example for openib_cma and
openib_scm<BR>+#<BR>+# For scm version you specify <ia_params> as
actual device name and port<BR>+# For cma version you specify
<ia_params> as:<BR>+# network
address, network hostname, or netdev name and 0 for
port<BR>+#<BR>+OpenIB-scm1 u1.2 nonthreadsafe default
/usr/local/openib_dapl/udapl/Target/libdapl.so mv_dapl.1.2 "mthca0 1"
""<BR>+OpenIB-scm2 u1.2 nonthreadsafe default
/usr/local/openib_dapl/udapl/Target/libdapl.so mv_dapl.1.2 "mthca0 2"
""<BR>+OpenIB-cma-ip u1.2 nonthreadsafe default
/usr/local/openib_dapl/udapl/Target/libdapl.so mv_dapl.1.2 "192.168.0.22 0"
""<BR>+OpenIB-cma-name u1.2 nonthreadsafe default
/usr/local/openib_dapl/udapl/Target/libdapl.so mv_dapl.1.2 "svr1-ib0 0"
""<BR>+OpenIB-cma-netdev u1.2 nonthreadsafe default
/usr/local/openib_dapl/udapl/Target/libdapl.so mv_dapl.1.2 "ib0 0"
""<BR> <BR>-IB1 u1.2 nonthreadsafe default <BR>Index:
test/dtest/makefile<BR>===================================================================<BR>---
test/dtest/makefile (revision 4589)<BR>+++ test/dtest/makefile (working
copy)<BR>@@ -4,13 +4,18 @@ CFLAGS = -O2
-g<BR> DAT_INC = ../../dat/include<BR> DAT_LIB =
/usr/local/lib<BR> <BR>-all: dtest<BR>+all: dtest
dtest_ext<BR> <BR> clean:<BR>- rm -f
*.o;touch *.c;rm -f dtest<BR>+ rm -f *.o;touch
*.c;rm -f dtest dtest_ext<BR> <BR> dtest:
./dtest.c<BR> $(CC) $(CFLAGS) ./dtest.c -o
dtest \<BR> -DDAPL_PROVIDER='"OpenIB-cma-ip"'
\<BR> -I $(DAT_INC) -L $(DAT_LIB)
-ldat<BR> <BR>+dtest_ext: ./dtest_ext.c<BR>+
$(CC) $(CFLAGS) ./dtest_ext.c -o dtest_ext \<BR>+
-DDAPL_PROVIDER='"OpenIB-cma-ip"' \<BR>+ -I
$(DAT_INC) -L $(DAT_LIB) -ldat<BR>+<BR>Index:
test/dtest/README<BR>===================================================================<BR>---
test/dtest/README (revision 4589)<BR>+++
test/dtest/README (working copy)<BR>@@ -1,10 +1,11
@@<BR> simple dapl test just for initial openIB uDAPL
testing...<BR> <BR>
dtest/dtest.c<BR>+
dtest/dtest_ext.c<BR>
dtest/makefile<BR>
dtest/dat.conf<BR> <BR>-to build (default uDAPL name == IB1, ib
device == mthca0, port == 1)<BR>+to build (default uDAPL name ==
OpenIB-cma-ip)<BR> edit makefile
and change path (DAT_LIB) to appropriate
libdat.so<BR> edit dat.conf and
change path to appropriate
libdapl.so<BR> cp dat.conf to
/etc/dat.conf<BR>Index:
dapl/include/dapl.h<BR>===================================================================<BR>---
dapl/include/dapl.h (revision 4589)<BR>+++ dapl/include/dapl.h (working
copy)<BR>@@ -1,25 +1,28 @@<BR> /*<BR>- * Copyright (c) 2002-2003,
Network Appliance, Inc. All rights reserved.<BR>+ * Copyright (c) 2002-2005,
Network Appliance, Inc. All rights reserved.<BR> *<BR> * This
Software is licensed under one of the following licenses:<BR>- *<BR>+ *
<BR> * 1) under the terms of the "Common Public License 1.0" a copy of
which is<BR>- * available from the Open Source Initiative,
see<BR>+ * in the file LICENSE.txt in the root directory.
The license is also<BR>+ * available from the Open Source
Initiative, see <BR> *
http://www.opensource.org/licenses/cpl.php.<BR>- *<BR>- * 2) under the terms
of the "The BSD License" a copy of which is<BR>- *
available from the Open Source Initiative, see<BR>+ * <BR>+ * 2) under the
terms of the "The BSD License" a copy of which is in the file<BR>+
* LICENSE2.txt in the root directory. The license is also
available from<BR>+ * the Open Source Initiative,
see<BR> *
http://www.opensource.org/licenses/bsd-license.php.<BR>- *<BR>+ * <BR>
* 3) under the terms of the "GNU General Public License (GPL) Version 2"
a<BR>- * copy of which is available from the Open Source
Initiative, see<BR>+ * copy of which is in the file
LICENSE3.txt in the root directory. The<BR>+ * license is
also available from the Open Source Initiative, see<BR>
* http://www.opensource.org/licenses/gpl-license.php.<BR>-
*<BR>+ * <BR> * Licensee has the right to choose one of the above
licenses.<BR>- *<BR>+ * <BR> * Redistributions of source code must
retain the above copyright<BR> * notice and one of the license
notices.<BR>- *<BR>+ * <BR> * Redistributions in binary form must
reproduce both the above copyright<BR> * notice, one of the license
notices in the documentation<BR> * and/or other materials provided
with the distribution.<BR>@@ -61,6 +64,8 @@<BR> #include
"dapl_dummy_util.h"<BR> #elif OPENIB<BR> #include
"dapl_ib_util.h"<BR>+#elif DET<BR>+#include
"dapl_det_util.h"<BR> #endif<BR> <BR> /*********************************************************************<BR>@@
-213,6 +218,10 @@ typedef struct
dapl_cookie
DAPL_COOKIE;<BR> typedef struct
dapl_dto_cookie
DAPL_DTO_COOKIE;<BR> typedef struct
dapl_rmr_cookie
DAPL_RMR_COOKIE;<BR> <BR>+#ifdef DAPL_EXTENSIONS<BR>+typedef struct
dapl_ext_cookie
DAPL_EXT_COOKIE;<BR>+#endif<BR>+<BR> typedef struct
dapl_private
DAPL_PRIVATE;<BR> <BR> typedef void
(*DAPL_CONNECTION_STATE_HANDLER) (<BR>@@ -563,6 +572,13 @@ typedef enum
dapl_dto_type<BR>
DAPL_DTO_TYPE_RECV,<BR>
DAPL_DTO_TYPE_RDMA_WRITE,<BR>
DAPL_DTO_TYPE_RDMA_READ,<BR>+#ifdef DAPL_EXTENSIONS<BR>+
DAPL_DTO_TYPE_RDMA_WRITE_IMMED,<BR>+
DAPL_DTO_TYPE_RECV_IMMED,<BR>+
DAPL_DTO_TYPE_CMP_AND_SWAP,<BR>+
DAPL_DTO_TYPE_FETCH_AND_ADD,<BR>+#endif<BR>+<BR> }
DAPL_DTO_TYPE;<BR> <BR> typedef enum dapl_cookie_type<BR>@@ -570,6
+586,9 @@ typedef enum dapl_cookie_type<BR>
DAPL_COOKIE_TYPE_NULL,<BR>
DAPL_COOKIE_TYPE_DTO,<BR>
DAPL_COOKIE_TYPE_RMR,<BR>+#ifdef DAPL_EXTENSIONS<BR>+
DAPL_COOKIE_TYPE_EXTENSION,<BR>+#endif<BR> }
DAPL_COOKIE_TYPE;<BR> <BR> /* DAPL_DTO_COOKIE used as context for
DTO WQEs */<BR>@@ -587,6 +606,27 @@ struct
dapl_rmr_cookie<BR>
DAT_RMR_COOKIE
cookie;<BR> };<BR> <BR>+#ifdef DAPL_EXTENSIONS<BR>+<BR>+/* DAPL
extended cookie types */<BR>+typedef enum
dapl_ext_type<BR>+{<BR>+
DAPL_EXT_TYPE_RDMA_WRITE_IMMED,<BR>+
DAPL_EXT_TYPE_CMP_AND_SWAP,<BR>+
DAPL_EXT_TYPE_FETCH_AND_ADD,<BR>+ DAPL_EXT_TYPE_RECV<BR>+}
DAPL_EXT_TYPE;<BR>+<BR>+/* DAPL extended cookie */<BR>+struct
dapl_ext_cookie<BR>+{<BR>+
DAPL_EXT_TYPE
type;<BR>+
DAT_DTO_COOKIE
cookie;<BR>+
DAT_COUNT
size; /* used RDMA write with immed
*/<BR>+};<BR>+<BR>+#endif<BR>+<BR> /* DAPL_COOKIE used as context for
WQEs */<BR> struct dapl_cookie<BR> {<BR>@@ -597,6 +637,9 @@ struct
dapl_cookie<BR> {<BR>
DAPL_DTO_COOKIE dto;<BR>
DAPL_RMR_COOKIE
rmr;<BR>+#ifdef DAPL_EXTENSIONS<BR>+
DAPL_EXT_COOKIE
ext;<BR>+#endif<BR> }
val;<BR> };<BR> <BR>@@ -1116,6 +1159,15 @@ extern DAT_RETURN
dapl_srq_set_lw(<BR>
IN
DAT_SRQ_HANDLE, /*
srq_handle
*/<BR> IN
DAT_COUNT);
/* low_watermark
*/<BR> <BR>+#ifdef DAPL_EXTENSIONS<BR>+<BR>+extern DAT_RETURN
dapl_extensions(<BR>+
IN
DAT_HANDLE, /*
dat_handle
*/<BR>+ IN DAT_EXT_OP,
/*
extension operation */<BR>+
IN va_list ); /*
va_list args
*/<BR>+<BR>+#endif<BR>+<BR> /*<BR> * DAPL internal utility
function prototpyes<BR> */<BR>Index:
dapl/udapl/Makefile<BR>===================================================================<BR>---
dapl/udapl/Makefile (revision 4589)<BR>+++ dapl/udapl/Makefile (working
copy)<BR>@@ -156,6 +156,7 @@ PROVIDER =
$(TOPDIR)/../openib_cma<BR> CFLAGS +=
-DOPENIB<BR> CFLAGS +=
-DCQ_WAIT_OBJECT<BR> CFLAGS +=
-I/usr/local/include/infiniband<BR>+CFLAGS +=
-I/usr/local/include/rdma<BR> endif<BR> <BR> #<BR>@@ -168,6
+169,12 @@ endif<BR> # VN_MEM_SHARED_VIRTUAL_SUPPORT <BR> #
CFLAGS += -DVN_MEM_SHARED_VIRTUAL_SUPPORT=1<BR> <BR>+# If
an implementation supports DAPL extensions<BR>+CFLAGS +=
-DDAPL_EXTENSIONS<BR>+<BR>+# If an implementation supports DAPL provider
specific attributes<BR>+CFLAGS +=
-DDAPL_PROVIDER_SPECIFIC_ATTR<BR>+<BR> CFLAGS += -I.
<BR> CFLAGS += -I.. <BR> CFLAGS +=
-I../../dat/include <BR>@@ -283,6 +290,8 @@ LDFLAGS += -libverbs
-lrdmacm<BR> LDFLAGS += -rpath /usr/local/lib -L
/usr/local/lib<BR> PROVIDER_SRCS = dapl_ib_util.c dapl_ib_cq.c
dapl_ib_qp.c
\<BR>
dapl_ib_cm.c dapl_ib_mem.c<BR>+# implementation supports DAPL
extensions<BR>+PROVIDER_SRCS +=
dapl_ib_extensions.c<BR> endif<BR> <BR> UDAPL_SRCS =
dapl_init.c
\<BR>Index:
dapl/common/dapl_ia_query.c<BR>===================================================================<BR>---
dapl/common/dapl_ia_query.c (revision 4589)<BR>+++
dapl/common/dapl_ia_query.c (working copy)<BR>@@
-167,6 +167,14 @@ dapl_ia_query (<BR> #if
!defined(__KDAPL__)<BR>
provider_attr->pz_support
= DAT_PZ_UNIQUE;<BR> #endif /* !KDAPL
*/<BR>+<BR>+ /*<BR>+ *
Have provider set their own.<BR>+ */<BR>+#ifdef
DAPL_PROVIDER_SPECIFIC_ATTR<BR>+
dapls_set_provider_specific_attr(provider_attr);<BR>+#endif<BR>+<BR>
/*<BR> * Set up
evd_stream_merging_supported options. Note there
is<BR> * one bit per allowable
combination, using the ordinal<BR>Index:
dapl/common/dapl_adapter_util.h<BR>===================================================================<BR>---
dapl/common/dapl_adapter_util.h (revision 4589)<BR>+++
dapl/common/dapl_adapter_util.h (working copy)<BR>@@ -256,6 +256,21 @@
dapls_ib_wait_object_wait (<BR> IN u_int32_t
timeout);<BR> #endif<BR> <BR>+#ifdef
DAPL_PROVIDER_SPECIFIC_ATTR<BR>+void
<BR>+dapls_set_provider_specific_attr(<BR>+ IN
DAT_PROVIDER_ATTR
*provider_attr );<BR>+#endif<BR>+<BR>+#ifdef
DAPL_EXTENSIONS<BR>+void<BR>+dapls_cqe_to_event_extension(<BR>+
IN
DAPL_EP
*ep_ptr,<BR>+ IN
DAPL_COOKIE
*cookie,<BR>+ IN
ib_work_completion_t
*cqe_ptr,<BR>+ OUT
DAT_EVENT
*event_ptr);<BR>+#endif<BR>+<BR> /*<BR> * Values for provider
DAT_NAMED_ATTR<BR> */<BR>@@ -272,6 +287,8 @@ dapls_ib_wait_object_wait
(<BR> #include "dapl_dummy_dto.h"<BR> #elif
OPENIB<BR> #include "dapl_ib_dto.h"<BR>+#elif DET<BR>+#include
"dapl_det_dto.h"<BR> #endif<BR> <BR> <BR>Index:
dapl/common/dapl_provider.c<BR>===================================================================<BR>---
dapl/common/dapl_provider.c (revision 4589)<BR>+++
dapl/common/dapl_provider.c (working copy)<BR>@@
-221,7 +221,11 @@ DAT_PROVIDER g_dapl_provider_template
=<BR>
&dapl_srq_post_recv,<BR>
&dapl_srq_query,<BR>
&dapl_srq_resize,<BR>-
&dapl_srq_set_lw<BR>+
&dapl_srq_set_lw,<BR>+<BR>+#ifdef DAPL_EXTENSIONS<BR>+
&dapl_extensions<BR>+#endif<BR> <BR> };<BR> #endif /*
__KDAPL__ */<BR>Index:
dapl/common/dapl_evd_util.c<BR>===================================================================<BR>---
dapl/common/dapl_evd_util.c (revision 4589)<BR>+++
dapl/common/dapl_evd_util.c (working copy)<BR>@@
-502,6 +502,20 @@ dapli_evd_eh_print_cqe (<BR> #ifdef
DAPL_DBG<BR> static char *optable[]
=<BR> {<BR>+#ifdef
OPENIB<BR>+ /* different order for openib verbs
*/<BR>+ "OP_RDMA_WRITE",<BR>+
"OP_RDMA_WRITE_IMM",<BR>+
"OP_SEND",<BR>+
"OP_SEND_IMM",<BR>+
"OP_RDMA_READ",<BR>+
"OP_COMP_AND_SWAP",<BR>+
"OP_FETCH_AND_ADD",<BR>+
"OP_RECEIVE",<BR>+
"OP_RECEIVE_IMM",<BR>+
"OP_BIND_MW",<BR>+
"OP_INVALID",<BR>+#else<BR>
"OP_SEND",<BR>
"OP_RDMA_READ",<BR> "OP_RDMA_WRITE",<BR>@@
-509,6 +523,7 @@ dapli_evd_eh_print_cqe (<BR>
"OP_FETCH_AND_ADD",<BR>
"OP_RECEIVE",<BR>
"OP_BIND_MW",<BR>+#endif<BR>
0<BR> };<BR> <BR>@@ -1113,6 +1128,15 @@
dapli_evd_cqe_to_event (<BR>
dapls_cookie_dealloc (&ep_ptr->req_buffer,
cookie);<BR>
break;<BR> }<BR>+<BR>+#ifdef
DAPL_EXTENSIONS<BR>+ case
DAPL_COOKIE_TYPE_EXTENSION:<BR>+
{<BR>+
dapls_cqe_to_event_extension(ep_ptr, cookie, cqe_ptr,
event_ptr);<BR>+
break;<BR>+
}<BR>+#endif<BR>+<BR>
default:<BR>
{<BR> dapl_os_assert
(!"Invalid Operation type");<BR>Index:
dapl/openib_cma/dapl_ib_dto.h<BR>===================================================================<BR>---
dapl/openib_cma/dapl_ib_dto.h (revision 4589)<BR>+++
dapl/openib_cma/dapl_ib_dto.h (working copy)<BR>@@ -35,7 +35,7
@@<BR> *<BR> * Description: <BR> *<BR>-
* The uDAPL openib provider - DTO operations and CQE macros
<BR>+ * The OpenIB uCMA provider - DTO operations and CQE macros
<BR> *<BR>
****************************************************************************<BR>
* Source
Control System Information<BR>@@ -119,7 +119,6 @@ dapls_ib_post_recv
(<BR> return
DAT_SUCCESS;<BR> }<BR> <BR>-<BR> /*<BR> *
dapls_ib_post_send<BR> *<BR>@@ -191,7 +190,7 @@ dapls_ib_post_send
(<BR> <BR> if (cookie != NULL)
<BR>
cookie->val.dto.size = total_len;<BR>-
<BR>+<BR> if ((op_type == OP_RDMA_WRITE) ||
(op_type == OP_RDMA_READ))
{<BR>
wr.wr.rdma.remote_addr =
remote_iov->target_address;<BR>
wr.wr.rdma.rkey = remote_iov->rmr_context;<BR>@@ -224,6 +223,152 @@
dapls_ib_post_send (<BR> return
DAT_SUCCESS;<BR> }<BR> <BR>+#ifdef DAPL_EXTENSIONS<BR>+/*<BR>+ *
dapls_ib_post_ext_send<BR>+ *<BR>+ * Provider specific extended Post SEND
function<BR>+ */<BR>+STATIC _INLINE_ DAT_RETURN <BR>+dapls_ib_post_ext_send
(<BR>+ IN
DAPL_EP
*ep_ptr,<BR>+ IN
ib_send_op_type_t
op_type,<BR>+ IN
DAPL_COOKIE
*cookie,<BR>+ IN
DAT_COUNT
segments,<BR>+ IN
DAT_LMR_TRIPLET
*local_iov,<BR>+ IN const
DAT_RMR_TRIPLET
*remote_iov,<BR>+ IN
DAT_UINT32
idata,<BR>+ IN
DAT_UINT64
compare_add,<BR>+ IN
DAT_UINT64
swap,<BR>+ IN
DAT_COMPLETION_FLAGS
completion_flags)<BR>+{<BR>+
dapl_dbg_log(DAPL_DBG_TYPE_EP,<BR>+
" post_snd: ep %p op %d ck %p
sgs",<BR>+
"%d l_iov %p r_iov %p f
%d\n",<BR>+
ep_ptr, op_type, cookie, segments, local_iov,
<BR>+
remote_iov,
completion_flags);<BR>+<BR>+ ib_data_segment_t
ds_array[DEFAULT_DS_ENTRIES];<BR>+ ib_data_segment_t
*ds_array_p;<BR>+ struct ibv_send_wr
wr;<BR>+ struct ibv_send_wr
*bad_wr;<BR>+ ib_hca_transport_t *ibt_ptr =
<BR>+
&ep_ptr->header.owner_ia->hca_ptr->ib_trans;<BR>+
DAT_COUNT i, total_len;<BR>+
<BR>+
dapl_dbg_log(DAPL_DBG_TYPE_EP,<BR>+
" post_snd: ep %p cookie %p segs %d l_iov
%p\n",<BR>+
ep_ptr, cookie, segments,
local_iov);<BR>+<BR>+ if(segments <=
DEFAULT_DS_ENTRIES)
<BR>+ ds_array_p
= ds_array;<BR>+
else<BR>+
ds_array_p =
<BR>+
dapl_os_alloc(segments *
sizeof(ib_data_segment_t));<BR>+<BR>+ if (NULL ==
ds_array_p)<BR>+
return (DAT_INSUFFICIENT_RESOURCES);<BR>+
<BR>+ /* setup the work request
*/<BR>+ wr.next = 0;<BR>+
wr.opcode = op_type;<BR>+ wr.num_sge =
0;<BR>+ wr.send_flags =
0;<BR>+ wr.wr_id =
(uint64_t)(uintptr_t)cookie;<BR>+ wr.sg_list =
ds_array_p;<BR>+ total_len =
0;<BR>+<BR>+ for (i = 0; i < segments; i++ )
{<BR>+ if (
!local_iov[i].segment_length
)<BR>+
continue;<BR>+<BR>+
ds_array_p->addr = (uint64_t)
local_iov[i].virtual_address;<BR>+
ds_array_p->length =
local_iov[i].segment_length;<BR>+
ds_array_p->lkey =
local_iov[i].lmr_context;<BR>+
<BR>+
dapl_dbg_log(DAPL_DBG_TYPE_EP,
<BR>+
" post_snd: lkey 0x%x va %p len
%d\n",<BR>+
ds_array_p->lkey, ds_array_p->addr,
<BR>+
ds_array_p->length
);<BR>+<BR>+
total_len +=
ds_array_p->length;<BR>+
wr.num_sge++;<BR>+
ds_array_p++;<BR>+
}<BR>+<BR>+ if (cookie != NULL)
<BR>+
cookie->val.dto.size = total_len;<BR>+<BR>+ if
((op_type == OP_RDMA_WRITE) || <BR>+
(op_type == OP_RDMA_WRITE_IMM)
||<BR>+ (op_type ==
OP_RDMA_READ))
{<BR>+
wr.wr.rdma.remote_addr =
remote_iov->target_address;<BR>+
wr.wr.rdma.rkey =
remote_iov->rmr_context;<BR>+
dapl_dbg_log(DAPL_DBG_TYPE_EP,
<BR>+
" post_snd_rdma: rkey 0x%x va
%#016Lx\n",<BR>+
wr.wr.rdma.rkey,
wr.wr.rdma.remote_addr);<BR>+
}<BR>+<BR>+ switch (op_type)
{<BR>+ case
OP_RDMA_WRITE_IMM:<BR>+
dapl_dbg_log(DAPL_DBG_TYPE_EP,
<BR>+
" post_snd: OP_RDMA_WRITE_IMMED=0x%x\n", idata
);<BR>+
wr.imm_data =
idata;<BR>+
break;<BR>+ case
OP_COMP_AND_SWAP:<BR>+
/* OP_COMP_AND_SWAP has direct IBAL wr_type mapping
*/<BR>+
dapl_dbg_log(DAPL_DBG_TYPE_EP,
<BR>+
" post_snd:
OP_COMP_AND_SWAP=%lx,"<BR>+
"%lx rkey 0x%x va
%#016Lx\n",<BR>+
compare_add, swap,
remote_iov->rmr_context,<BR>+
remote_iov->target_address);<BR>+
<BR>+
wr.wr.atomic.compare_add =
compare_add;<BR>+
wr.wr.atomic.swap =
swap;<BR>+
wr.wr.atomic.remote_addr =
remote_iov->target_address;<BR>+
wr.wr.atomic.rkey =
remote_iov->rmr_context;<BR>+
break;<BR>+ case
OP_FETCH_AND_ADD:<BR>+
/* OP_FETCH_AND_ADD has direct IBAL wr_type mapping
*/<BR>+
dapl_dbg_log(DAPL_DBG_TYPE_EP,
<BR>+
" post_snd:
OP_FETCH_AND_ADD=%lx,"<BR>+
"%lx rkey 0x%x va
%#016Lx\n",<BR>+
compare_add,
remote_iov->rmr_context,<BR>+
remote_iov->target_address);<BR>+<BR>+
wr.wr.atomic.compare_add =
compare_add;<BR>+
wr.wr.atomic.remote_addr =
remote_iov->target_address;<BR>+
wr.wr.atomic.rkey =
remote_iov->rmr_context;<BR>+
break;<BR>+
default:<BR>+
break;<BR>+ }<BR>+<BR>+ /*
inline data for send or write ops */<BR>+ if
((total_len <= ibt_ptr->max_inline_send) &&
<BR>+ ((op_type == OP_SEND) || (op_type
== OP_RDMA_WRITE)))
<BR>+
wr.send_flags |= IBV_SEND_INLINE;<BR>+
<BR>+ /* set completion flags in work request
*/<BR>+ wr.send_flags |=
(DAT_COMPLETION_SUPPRESS_FLAG &
<BR>+
completion_flags) ? 0 : IBV_SEND_SIGNALED;<BR>+
wr.send_flags |= (DAT_COMPLETION_BARRIER_FENCE_FLAG &
<BR>+
completion_flags) ? IBV_SEND_FENCE : 0;<BR>+
wr.send_flags |= (DAT_COMPLETION_SOLICITED_WAIT_FLAG &
<BR>+
completion_flags) ? IBV_SEND_SOLICITED :
0;<BR>+<BR>+ dapl_dbg_log(DAPL_DBG_TYPE_EP,
<BR>+
" post_snd: op 0x%x flags 0x%x sglist %p, %d\n",
<BR>+
wr.opcode, wr.send_flags, wr.sg_list,
wr.num_sge);<BR>+<BR>+ if
(ibv_post_send(ep_ptr->qp_handle->cm_id->qp, &wr,
&bad_wr))<BR>+
return( dapl_convert_errno(EFAULT,"ibv_recv")
);<BR>+ <BR>+
dapl_dbg_log(DAPL_DBG_TYPE_EP," post_snd:
returned\n");<BR>+ return
DAT_SUCCESS;<BR>+}<BR>+#endif<BR>+<BR> STATIC _INLINE_ DAT_RETURN
<BR> dapls_ib_optional_prv_dat(<BR>
IN DAPL_CR *cr_ptr,<BR>Index:
dapl/openib_cma/dapl_ib_util.c<BR>===================================================================<BR>---
dapl/openib_cma/dapl_ib_util.c (revision 4589)<BR>+++
dapl/openib_cma/dapl_ib_util.c (working copy)<BR>@@ -35,7 +35,7
@@<BR> *<BR> * Description: <BR> *<BR>-
* The uDAPL openib provider - init, open, close, utilities, work
thread<BR>+ * The OpenIB uCMA provider - init, open, close,
utilities, work thread<BR> *<BR>
****************************************************************************<BR>
* Source
Control System Information<BR>@@ -64,7 +64,6 @@ static const char rcsid[] =
"$Id: $";<BR> #include <net/if.h>
/* for struct ifreq */<BR> #include <net/if_arp.h> /* for
ARPHRD_INFINIBAND */<BR> <BR>-<BR> int g_dapl_loopback_connection
= 0;<BR> int g_ib_pipe[2];<BR> ib_thread_state_t g_ib_thread_state
= 0;<BR>@@ -727,7 +726,7 @@ void dapli_thread(void *arg)
<BR>
int
ret,idx,fds;<BR>
char
rbuf[2];<BR> <BR>-
dapl_dbg_log (DAPL_DBG_TYPE_CM,<BR>+ dapl_dbg_log
(DAPL_DBG_TYPE_UTIL,<BR>
" ib_thread(%d,0x%x): ENTER: pipe %d ucma
%d\n",<BR>
getpid(), g_ib_thread, g_ib_pipe[0],
rdma_get_fd());<BR> <BR>@@ -767,7 +766,7 @@ void dapli_thread(void
*arg)
<BR>
ufds[idx].revents = 0;<BR>
uhca[idx]
=
hca;<BR> <BR>-
dapl_dbg_log(DAPL_DBG_TYPE_CM,<BR>+
dapl_dbg_log(DAPL_DBG_TYPE_UTIL,<BR>
" ib_thread(%d) poll_fd: hca[%d]=%p,
async=%d"<BR>
" pipe=%d cm=%d
cq=d\n",<BR>
getpid(), hca, ufds[idx-1].fd, <BR>@@ -783,14 +782,14 @@ void
dapli_thread(void *arg)
<BR>
dapl_os_unlock(&g_hca_lock);<BR>
ret = poll(ufds, fds, -1);
<BR> if
(ret <= 0)
{<BR>-
dapl_dbg_log(DAPL_DBG_TYPE_WARN,<BR>+
dapl_dbg_log(DAPL_DBG_TYPE_UTIL,<BR>
" ib_thread(%d): ERR %s
poll\n",<BR>
getpid(),strerror(errno));<BR>
dapl_os_lock(&g_hca_lock);<BR>
continue;<BR>
}<BR> <BR>-
dapl_dbg_log(DAPL_DBG_TYPE_CM,<BR>+
dapl_dbg_log(DAPL_DBG_TYPE_UTIL,<BR>
" ib_thread(%d) poll_event:
"<BR>
" async=0x%x pipe=0x%x cm=0x%x
cq=0x%x\n",<BR>
getpid(), ufds[idx-1].revents, ufds[0].revents, <BR>@@ -834,3 +833,63 @@
void dapli_thread(void *arg) <BR>
dapl_os_unlock(&g_hca_lock); <BR> }<BR> <BR>+#ifdef
DAPL_PROVIDER_SPECIFIC_ATTR<BR>+/*<BR>+ *
dapls_set_provider_specific_attr<BR>+ *<BR>+ * Input:<BR>+ *
attr_ptr Pointer provider attributes<BR>+ *<BR>+ *
Output:<BR>+ * none<BR>+ *<BR>+ * Returns:<BR>+ * void<BR>+
*/<BR>+DAT_NAMED_ATTR ib_attrs[] = {<BR>+ <BR>+#ifdef
DAPL_EXTENSIONS<BR>+ {<BR>+
DAT_EXT_ATTR,<BR>+
DAT_EXT_ATTR_TRUE<BR>+ },<BR>+
{<BR>+
DAT_EXT_ATTR_RDMA_WRITE_IMMED,<BR>+
DAT_EXT_ATTR_TRUE<BR>+ },<BR>+
{<BR>+ DAT_EXT_ATTR_RECV_IMMED,<BR>+
DAT_EXT_ATTR_TRUE<BR>+ },<BR>+ /*
inbound immediate data placed in event, NOT payload
*/<BR>+ {<BR>+
DAT_EXT_ATTR_RECV_IMMED_EVENT,<BR>+
DAT_EXT_ATTR_TRUE<BR>+ },<BR>+
{<BR>+ DAT_EXT_ATTR_FETCH_AND_ADD,<BR>+
DAT_EXT_ATTR_TRUE<BR>+ },<BR>+
{<BR>+ DAT_EXT_ATTR_CMP_AND_SWAP,<BR>+
DAT_EXT_ATTR_TRUE<BR>+
},<BR>+#else<BR>+ {<BR>+
"DAT_EXTENSION_INTERFACE",<BR>+
"FALSE"<BR>+ },<BR>+#endif<BR>+};<BR>+<BR>+#define
SPEC_ATTR_SIZE(x) ( sizeof(x)/sizeof(DAT_NAMED_ATTR) )<BR>+<BR>+void
dapls_set_provider_specific_attr(<BR>+ IN
DAT_PROVIDER_ATTR *attr_ptr )<BR>+{<BR>+
attr_ptr->num_provider_specific_attr =
SPEC_ATTR_SIZE(ib_attrs);<BR>+
attr_ptr->provider_specific_attr =
ib_attrs;<BR>+}<BR>+<BR>+#endif<BR>+<BR>Index:
dapl/openib_cma/dapl_ib_mem.c<BR>===================================================================<BR>---
dapl/openib_cma/dapl_ib_mem.c (revision 4589)<BR>+++
dapl/openib_cma/dapl_ib_mem.c (working copy)<BR>@@ -25,9 +25,9
@@<BR> <BR> /**********************************************************************<BR>
* <BR>- * MODULE: dapl_det_mem.c<BR>+ * MODULE: dapl_ib_mem.c<BR>
*<BR>- * PURPOSE: Intel DET APIs: Memory windows, registration,<BR>+ *
PURPOSE: OpenIB uCMA provider Memory windows, registration,<BR>
* and protection
domain <BR> *<BR> * $Id: $<BR>@@ -72,12 +72,10 @@
dapls_convert_privileges(IN
DAT_MEM_PRIV<BR>
access |= IBV_ACCESS_LOCAL_WRITE;<BR> if
(DAT_MEM_PRIV_REMOTE_WRITE_FLAG &
privileges)<BR>
access |= IBV_ACCESS_REMOTE_WRITE;<BR>- if
(DAT_MEM_PRIV_REMOTE_READ_FLAG &
privileges)<BR>-
access |= IBV_ACCESS_REMOTE_READ;<BR>- if
(DAT_MEM_PRIV_REMOTE_READ_FLAG &
privileges)<BR>-
access |= IBV_ACCESS_REMOTE_READ;<BR>- if
(DAT_MEM_PRIV_REMOTE_READ_FLAG &
privileges)<BR>+ if (DAT_MEM_PRIV_REMOTE_READ_FLAG
& privileges)
{<BR>
access |=
IBV_ACCESS_REMOTE_READ;<BR>+
access |= IBV_ACCESS_REMOTE_ATOMIC;<BR>+
}<BR> <BR> return
access;<BR> }<BR>Index:
dapl/openib_cma/dapl_ib_cm.c<BR>===================================================================<BR>---
dapl/openib_cma/dapl_ib_cm.c (revision 4589)<BR>+++
dapl/openib_cma/dapl_ib_cm.c (working copy)<BR>@@ -35,7
+35,7 @@<BR> *<BR> * Description: <BR> *<BR>-
* The uDAPL openib provider - connection management<BR>+
* The OpenIB uCMA provider - uCMA connection
management<BR> *<BR>
****************************************************************************<BR>
* Source
Control System Information<BR>@@ -592,7 +592,11 @@
dapls_ib_setup_conn_listener(IN DAPL_IA
<BR> <BR> if
(rdma_bind_addr(conn->cm_id,<BR>
(struct sockaddr *)&ia_ptr->hca_ptr->hca_address))
{<BR>-
dat_status =
dapl_convert_errno(errno,"setup_listener");<BR>+
if (errno ==
-EBUSY)<BR>+
dat_status =
DAT_CONN_QUAL_IN_USE;<BR>+
else<BR>+
dat_status =
<BR>+
dapl_convert_errno(errno,"setup_listener");<BR>
goto bail;<BR> }<BR> <BR>Index:
dapl/openib_cma/dapl_ib_qp.c<BR>===================================================================<BR>---
dapl/openib_cma/dapl_ib_qp.c (revision 4589)<BR>+++
dapl/openib_cma/dapl_ib_qp.c (working copy)<BR>@@ -25,9
+25,9
@@<BR> <BR> /**********************************************************************<BR>
*<BR>- * MODULE: dapl_det_qp.c<BR>+ * MODULE: dapl_ib_qp.c<BR> *<BR>-
* PURPOSE: QP routines for access to DET Verbs<BR>+ * PURPOSE: OpenIB uCMA
QP routines <BR> *<BR> * $Id: $<BR>
**********************************************************************/<BR>Index:
dapl/openib_cma/dapl_ib_util.h<BR>===================================================================<BR>---
dapl/openib_cma/dapl_ib_util.h (revision 4589)<BR>+++
dapl/openib_cma/dapl_ib_util.h (working copy)<BR>@@ -35,7 +35,7
@@<BR> *<BR> * Description: <BR> *<BR>-
* The uDAPL openib provider - definitions, prototypes,<BR>+
* The OpenIB uCMA provider - definitions, prototypes,<BR>
*<BR>
****************************************************************************<BR>
* Source
Control System Information<BR>Index:
dapl/openib_cma/README<BR>===================================================================<BR>---
dapl/openib_cma/README (revision 4589)<BR>+++
dapl/openib_cma/README (working copy)<BR>@@ -23,15 +23,22
@@ New files for openib_scm provider<BR>
dapl/openib_cma/dapl_ib_util.c<BR>
dapl/openib_cma/dapl_ib_util.h<BR>
dapl/openib_cma/dapl_ib_cm.c<BR>+
dapl/openib_cma/dapl_ib_extensions.c<BR> <BR> A simple dapl test
just for openib_scm testing...<BR> <BR>
test/dtest/dtest.c<BR>+
test/dtest/dtest_ext.c<BR>
test/dtest/makefile<BR> <BR>
server: dtest -s <BR>
client: dtest -h hostname<BR> <BR>+or with
extensions<BR>+<BR>+ server:
dtest_ext -s <BR>+ client:
dtest_ext -h hostname<BR>+<BR> known
issues:<BR> <BR> no memory windows
support in ibverbs, dat_create_rmr fails.<BR>Index:
dapl/openib_cma/dapl_ib_cq.c<BR>===================================================================<BR>---
dapl/openib_cma/dapl_ib_cq.c (revision 4589)<BR>+++
dapl/openib_cma/dapl_ib_cq.c (working copy)<BR>@@ -35,7
+35,7 @@<BR> *<BR> * Description: <BR> *<BR>-
* The uDAPL openib provider - completion queue<BR>+
* The OpenIB uCMA provider - completion queue<BR>
*<BR>
****************************************************************************<BR>
* Source
Control System Information<BR>@@ -498,7 +498,10 @@
dapls_ib_wait_object_wait(IN ib_wait_obj<BR>
if (timeout !=
DAT_TIMEOUT_INFINITE)<BR>
timeout_ms = timeout/1000;<BR> <BR>- status =
poll(&cq_fd, 1, timeout_ms);<BR>+ /* restart
syscall */<BR>+ while ((status = poll(&cq_fd, 1,
timeout_ms)) == -1
)<BR>+ if (errno
==
EINTR)<BR>+
continue;<BR> <BR> /* returned event
*/<BR> if (status > 0) {<BR>@@ -511,6
+514,8 @@ dapls_ib_wait_object_wait(IN
ib_wait_obj<BR> /* timeout
*/<BR> } else if (status == 0)
<BR>
status = ETIMEDOUT;<BR>+ else
<BR>+ status =
errno;<BR> <BR>
dapl_dbg_log(DAPL_DBG_TYPE_UTIL,
<BR>
" cq_object_wait: RET evd %p ibv_cq %p ibv_ctx %p
%s\n",<BR>Index:
dat/include/dat/dat_redirection.h<BR>===================================================================<BR>---
dat/include/dat/dat_redirection.h (revision
4589)<BR>+++ dat/include/dat/dat_redirection.h
(working copy)<BR>@@ -59,10 +59,10 @@ typedef struct dat_provider
DAT_PROVIDER<BR> * This would allow a good compiler to avoid
indirection overhead when<BR> * making function calls.<BR>
*/<BR>-<BR> #define DAT_HANDLE_TO_PROVIDER(handle) (*(DAT_PROVIDER
**)(handle))<BR> #endif<BR> <BR>+<BR> #define
DAT_IA_QUERY(ia, evd, ia_msk, ia_ptr, p_msk, p_ptr)
\<BR> (*DAT_HANDLE_TO_PROVIDER
(ia)->ia_query_func)
(\<BR>
(ia),
\<BR>@@ -395,6 +395,12 @@ typedef struct dat_provider
DAT_PROVIDER<BR>
(lbuf),
\<BR>
(cookie))<BR> <BR>+#define DAT_EXTENSION(handle, op, args)
\<BR>+ (*DAT_HANDLE_TO_PROVIDER
(handle)->extension_func)
(\<BR>+
(handle), \<BR>+
(op), \<BR>+
(args))<BR>+<BR> /***************************************************************<BR>
*<BR> * FUNCTION PROTOTYPES<BR>@@ -720,4 +726,11 @@ typedef DAT_RETURN
(*DAT_SRQ_POST_RECV_F<BR>
IN DAT_LMR_TRIPLET *, /*
local_iov
*/<BR> IN
DAT_DTO_COOKIE ); /*
user_cookie
*/<BR> <BR>+/* Extension function */<BR>+#include
<stdarg.h><BR>+typedef DAT_RETURN (*DAT_EXTENSION_FUNC)
(<BR>+ IN
DAT_HANDLE, /* dat
handle
*/<BR>+ IN
DAT_EXT_OP, /* extension operation
*/<BR>+ IN va_list
); /*
va_list */<BR>+<BR> #endif /*
_DAT_REDIRECTION_H_ */<BR>Index:
dat/include/dat/dat.h<BR>===================================================================<BR>---
dat/include/dat/dat.h (revision 4589)<BR>+++
dat/include/dat/dat.h (working copy)<BR>@@ -854,11
+854,15 @@ typedef enum dat_event_number<BR>
DAT_ASYNC_ERROR_EP_BROKEN
= 0x08003,<BR>
DAT_ASYNC_ERROR_TIMED_OUT
=
0x08004,<BR>
DAT_ASYNC_ERROR_PROVIDER_INTERNAL_ERROR =
0x08005,<BR>-
DAT_SOFTWARE_EVENT
= 0x10001<BR>+
DAT_SOFTWARE_EVENT
= 0x10001,<BR>+
DAT_EXTENSION_EVENT
= 0x20001<BR>+<BR> } DAT_EVENT_NUMBER;<BR> <BR>-/* <st1:place
w:st="on">Union</st1:place> for event Data */<BR>+/* include extension data
definitions */<BR>+#include <dat/dat_extensions.h><BR> <BR>+/*
<st1:place w:st="on">Union</st1:place> for event Data */<BR> typedef
union dat_event_data<BR> {<BR>
DAT_DTO_COMPLETION_EVENT_DATA
dto_completion_event_data;<BR>@@ -867,6 +871,7 @@ typedef union
dat_event_data<BR>
DAT_CONNECTION_EVENT_DATA
connect_event_data;<BR>
DAT_ASYNCH_ERROR_EVENT_DATA
asynch_error_event_data;<BR>
DAT_SOFTWARE_EVENT_DATA
software_event_data;<BR>+
DAT_EXTENSION_DATA
extension_data;<BR> } DAT_EVENT_DATA;<BR> <BR> /* Event
struct that holds all event information */<BR>@@ -1222,6 +1227,11 @@ extern
DAT_RETURN dat_srq_set_lw (<BR>
IN
DAT_SRQ_HANDLE, /*
srq_handle
*/<BR> IN
DAT_COUNT);
/* low_watermark
*/<BR> <BR>+extern DAT_RETURN
dat_extension(<BR>+ IN
DAT_HANDLE,<BR>+ IN
DAT_EXT_OP,<BR>+ IN ...
);<BR>+<BR> /*<BR> * DAT registry functions.<BR>
*<BR>Index:
dat/include/dat/udat_redirection.h<BR>===================================================================<BR>---
dat/include/dat/udat_redirection.h (revision 4589)<BR>+++
dat/include/dat/udat_redirection.h (working copy)<BR>@@
-199,7 +199,6 @@ typedef DAT_RETURN (*DAT_EVD_SET_UNWAITA<BR> typedef
DAT_RETURN (*DAT_EVD_CLEAR_UNWAITABLE_FUNC)
(<BR> IN
DAT_EVD_HANDLE);
/* evd_handle
*/<BR> <BR>-<BR> #include
<dat/dat_redirection.h><BR> <BR> struct dat_provider<BR>@@
-294,6 +293,10 @@ struct dat_provider<BR>
DAT_SRQ_QUERY_FUNC
srq_query_func;<BR>
DAT_SRQ_RESIZE_FUNC
srq_resize_func;<BR>
DAT_SRQ_SET_LW_FUNC
srq_set_lw_func;<BR>+<BR>+ /* extension for provder
specific functions */<BR>+
DAT_EXTENSION_FUNC
extension_func;<BR>+<BR> };<BR> <BR> #endif /*
_UDAT_REDIRECTION_H_ */<BR>Index:
dat/include/dat/dat_extensions.h<BR>===================================================================<BR>---
dat/include/dat/dat_extensions.h (revision
0)<BR>+++ dat/include/dat/dat_extensions.h
(revision 0)<BR>@@ -0,0 +1,209 @@<BR>+/*<BR>+ * Copyright (c) 2002-2005,
Network Appliance, Inc. All rights reserved.<BR>+ *<BR>+ * This Software is
licensed under one of the following licenses:<BR>+ * <BR>+ * 1) under the
terms of the "Common Public License 1.0" a copy of which is<BR>+
* in the file LICENSE.txt in the root directory. The
license is also<BR>+ * available from the Open Source
Initiative, see <BR>+ *
http://www.opensource.org/licenses/cpl.php.<BR>+ * <BR>+ * 2) under the
terms of the "The BSD License" a copy of which is in the file<BR>+
* LICENSE2.txt in the root directory. The license is also
available from<BR>+ * the Open Source Initiative, see<BR>+
* http://www.opensource.org/licenses/bsd-license.php.<BR>+
* <BR>+ * 3) under the terms of the "GNU General Public License (GPL)
Version 2" a<BR>+ * copy of which is in the file
LICENSE3.txt in the root directory. The<BR>+ * license is
also available from the Open Source Initiative, see<BR>+ *
http://www.opensource.org/licenses/gpl-license.php.<BR>+ * <BR>+ * Licensee
has the right to choose one of the above licenses.<BR>+ * <BR>+ *
Redistributions of source code must retain the above copyright<BR>+ * notice
and one of the license notices.<BR>+ * <BR>+ * Redistributions in binary
form must reproduce both the above copyright<BR>+ * notice, one of the
license notices in the documentation<BR>+ * and/or other materials provided
with the distribution.<BR>+
*/<BR>+/**********************************************************************<BR>+
*<BR>+ * HEADER: dat_extensions.h<BR>+ *<BR>+ * PURPOSE: defines the
extensions to the DAT API for uDAPL.<BR>+ *<BR>+ * Description: Header file
for "uDAPL: User Direct Access Programming<BR>+
* Library, Version:
1.2"<BR>+ *<BR>+ * Mapping rules:<BR>+ * All
global symbols are prepended with "DAT_" or "dat_"<BR>+
* All DAT objects have an 'api' tag which,
such as 'ep' or 'lmr'<BR>+ * The method table
is in the provider definition structure.<BR>+ *<BR>+ *<BR>+
**********************************************************************/<BR>+<BR>+#ifndef
_DAT_EXTENSIONS_H_<BR>+<BR>+extern int dat_extensions;<BR>+<BR>+/* <BR>+ *
Provider specific attribute strings for extension support <BR>+
* returned with dat_ia_query() and <BR>+ *
DAT_PROVIDER_ATTR_MASK == DAT_PROVIDER_FIELD_PROVIDER_SPECIFIC_ATTR<BR>+
*<BR>+ * DAT_NAMED_ATTR name == extended
operation, <BR>+
*
value == TRUE if extended operation is supported<BR>+ */<BR>+#define
DAT_EXT_ATTR
"DAT_EXTENSION_INTERFACE"<BR>+#define
DAT_EXT_ATTR_RDMA_WRITE_IMMED
"DAT_EXT_RDMA_WRITE_IMMED"<BR>+#define
DAT_EXT_ATTR_RECV_IMMED
"DAT_EXT_RECV_IMMED"<BR>+#define
DAT_EXT_ATTR_RECV_IMMED_EVENT
"DAT_EXT_RECV_IMMED_EVENT"<BR>+#define DAT_EXT_ATTR_RECV_IMMED_PAYLOAD
"DAT_EXT_RECV_IMMED_PAYLOAD"<BR>+#define DAT_EXT_ATTR_FETCH_AND_ADD
"DAT_EXT_FETCH_AND_ADD"<BR>+#define DAT_EXT_ATTR_CMP_AND_SWAP
"DAT_EXT_CMP_AND_SWAP"<BR>+#define
DAT_EXT_ATTR_TRUE
"TRUE"<BR>+#define
DAT_EXT_ATTR_FALSE
"FALSE"<BR>+<BR>+/* <BR>+ * Extension OPERATIONS <BR>+ */<BR>+typedef enum
dat_ext_op<BR>+{<BR>+
DAT_EXT_RDMA_WRITE_IMMED,<BR>+
DAT_EXT_RECV_IMMED,<BR>+
DAT_EXT_FETCH_AND_ADD,<BR>+
DAT_EXT_CMP_AND_SWAP,<BR>+<BR>+} DAT_EXT_OP;<BR>+<BR>+/* <BR>+ * Extension
completion event TYPES<BR>+ */<BR>+typedef enum
dat_ext_event_type<BR>+{<BR>+
DAT_EXT_RDMA_WRITE_IMMED_STATUS = 1,<BR>+
DAT_EXT_RECV_NO_IMMED,<BR>+
DAT_EXT_RECV_IMMED_DATA_EVENT,
<BR>+
DAT_EXT_RECV_IMMED_DATA_PAYLOAD,
<BR>+
DAT_EXT_FETCH_AND_ADD_STATUS,
<BR>+
DAT_EXT_CMP_AND_SWAP_STATUS,
<BR>+<BR>+} DAT_EXT_EVENT_TYPE;<BR>+<BR>+/* <BR>+ * Extension completion
event DATA<BR>+ */<BR>+typedef struct dat_immediate_data
<BR>+{<BR>+ DAT_UINT32 data;<BR>+<BR>+}
DAT_RDMA_WRITE_IMMED_DATA;<BR>+<BR>+typedef struct
dat_extension_data<BR>+{<BR>+
DAT_DTO_COMPLETION_EVENT_DATA dto;<BR>+
DAT_EXT_EVENT_TYPE
type;<BR>+ union
{<BR>+
DAT_RDMA_WRITE_IMMED_DATA
immed; <BR>+ }
val;<BR>+} DAT_EXTENSION_DATA;<BR>+<BR>+typedef enum
dat_ext_flags<BR>+{<BR>+
DAT_EXT_WRITE_IMMED_FLAG =
0x1,<BR>+
DAT_EXT_WRITE_CONFIRM_FLAG = 0x2,<BR>+<BR>+}
DAT_EXT_FLAGS;<BR>+<BR>+/* <BR>+ * Extended API with redirection via DAT
extension function<BR>+ */<BR>+<BR>+/* <BR>+ * RDMA Write with IMMEDIATE
extension:<BR>+ *<BR>+ * Asynchronous call performs a normal RDMA write to
the remote endpoint <BR>+ * followed by a post of an extended immediate data
value to the receive <BR>+ * EVD on the remote endpoint. Event completion
for the request completes <BR>+ * as an DAT_EXTENSION_EVENT with type set to
DAT_EXT_RDMA_WRITE_IMMED_STATUS.<BR>+ * Event completion on the remote
endpoint completes as an DAT_EXTENSION_EVENT<BR>+ * with type set to
DAT_EXT_RECV_IMMED_DATA_IN_EVENT or <BR>+ *
DAT_EXT_RECV_IMMED_DATA_IN_PAYLOAD depending on the provider transport.<BR>+
*<BR>+ * DAT_EXT_WRITE_IMMED_FLAG requests that the supplied<BR>+
*'immediate' value be sent as the payload of a four byte send following<BR>+
* the RDMA Write, or any transport-dependent equivalent thereof.<BR>+ * For
example, on InfiniBand the request should be translated as an<BR>+ * RDMA
Write with Immediate. <BR>+ *<BR>+ * DAT_EXT_WRITE_CONFIRM_FLAG requests
that this DTO<BR>+ * not complete until receipt by the far end is confirmed.
<BR>+ *<BR>+ * Note to Consumers: the immediate data will consume a
receive<BR>+ * buffer at the Data Sink. <BR>+ *<BR>+ * Other extension
flags:<BR>+ * n/a<BR>+ */<BR>+#define
dat_ep_post_rdma_write_immed(ep, size, lbuf, cookie, rbuf, idata, eflgs,
flgs) \<BR>+
dat_extension( ep,
\<BR>+
DAT_EXT_RDMA_WRITE_IMMED,
\<BR>+
(size),
\<BR>+
(lbuf),
\<BR>+
(cookie),
\<BR>+
(rbuf),
\<BR>+
(idata),
\<BR>+
(eflgs),
\<BR>+
(flgs))<BR>+ <BR>+/* <BR>+ * Call performs a normal
post receive message to the local endpoint<BR>+ * that includes additional
32-bit buffer space for immediate data <BR>+ * Event completion for the
request completes as an <BR>+ * DAT_EXTENSION_EVENT with type set to
DAT_EXT_RDMA_WRITE_IMMED_STATUS.<BR>+ */<BR>+#define
dat_ep_post_recv_immed(ep, size, lbuf, cookie, flgs)
\<BR>+
dat_extension( ep,
\<BR>+
DAT_EXT_RECV_IMMED,
\<BR>+
(size),
\<BR>+
(lbuf),
\<BR>+
(cookie),
\<BR>+
(flgs))<BR>+<BR>+/*<BR>+ * This asynchronous call is modeled after the
InfiniBand atomic <BR>+ * Fetch and Add operation. The add_value is added to
the 64 bit <BR>+ * value stored at the remote memory location specified in
remote_iov<BR>+ * and the result is stored in the local_iov. <BR>+
*/<BR>+#define dat_ep_post_fetch_and_add(ep, add_val, lbuf, cookie, rbuf,
flgs) \<BR>+
dat_extension( ep,
\<BR>+
DAT_EXT_FETCH_AND_ADD,
\<BR>+
(add_val),
\<BR>+
(lbuf),
\<BR>+
(cookie),
\<BR>+
(rbuf),
\<BR>+
(flgs))<BR>+
<BR>+/*<BR>+ * This asynchronous call is modeled after the InfiniBand atomic
<BR>+ * Compare and Swap operation. The cmp_value is compared to the 64 bit
<BR>+ * value stored at the remote memory location specified in
remote_iov. <BR>+ * If the two values are equal, the 64 bit swap_value
is stored in <BR>+ * the remote memory location. In all cases, the
original 64 bit <BR>+ * value stored in the remote memory location is copied
to the local_iov.<BR>+ */<BR>+#define dat_ep_post_cmp_and_swap(ep, cmp_val,
swap_val, lbuf, cookie, rbuf, flgs)
\<BR>+
dat_extension( ep,
\<BR>+
DAT_EXT_CMP_AND_SWAP,
\<BR>+
(cmp_val),
\<BR>+
(swap_val),
\<BR>+
(lbuf),
\<BR>+
(cookie),
\<BR>+
(rbuf),
\<BR>+
(flgs))<BR>+<BR>+#endif /* _DAT_EXTENSIONS_H_ */<BR>+<BR>Index:
dat/common/dat_api.c<BR>===================================================================<BR>---
dat/common/dat_api.c (revision 4594)<BR>+++
dat/common/dat_api.c (working copy)<BR>@@
-1142,6 +1142,36 @@ DAT_RETURN
dat_srq_set_lw(<BR>
low_watermark);<BR> }<BR> <BR>+DAT_RETURN
dat_extension(<BR>+
IN DAT_HANDLE
handle,<BR>+
IN DAT_EXT_OP
ext_op,<BR>+
IN ...
)<BR>+<BR>+{<BR>+ DAT_RETURN
status;<BR>+ va_list
args;<BR>+<BR>+ if (handle ==
NULL)<BR>+
{<BR>+ return
DAT_ERROR(DAT_INVALID_HANDLE,
DAT_INVALID_HANDLE_EP);<BR>+
}<BR>+<BR>+ /* verify provider extension support
*/<BR>+ if
(!dat_extensions)<BR>+
{<BR>+ return
DAT_ERROR(DAT_NOT_IMPLEMENTED, 0);<BR>+
}<BR>+<BR>+ va_start(args,
ext_op);<BR>+<BR>+ status =
DAT_EXTENSION(handle,<BR>+
ext_op,<BR>+
args);<BR>+
va_end(args);<BR>+<BR>+ return
status;<BR>+}<BR>+<BR> /*<BR> * Local variables:<BR>
* c-indent-level: 4<BR>Index:
dat/udat/udat.c<BR>===================================================================<BR>---
dat/udat/udat.c (revision 4594)<BR>+++
dat/udat/udat.c (working copy)<BR>@@ -66,6 +66,10 @@
udat_check_state ( void );<BR>
*
*<BR>
*********************************************************************/<BR> <BR>+/*<BR>+
* Use a global to get an unresolved when run with pre-extension
library<BR>+ */<BR>+int dat_extensions = 0;<BR> <BR> /*<BR>
*<BR>@@ -230,13 +234,44 @@ dat_ia_openv
(<BR>
async_event_qlen,<BR>
async_event_handle,<BR>
ia_handle);<BR>+<BR>+
/*<BR>+ * See if provider supports
extensions<BR>+ */<BR> if
(dat_status == DAT_SUCCESS)<BR>
{<BR>- return_handle = dats_set_ia_handle
(*ia_handle);<BR>- if (return_handle >=
0)<BR>- {<BR>-
*ia_handle =
(DAT_IA_HANDLE)return_handle;<BR>-
}<BR>+
DAT_PROVIDER_ATTR
p_attr;<BR>+
int
i;<BR>+<BR>+ return_handle =
dats_set_ia_handle
(*ia_handle);<BR>+ if
(return_handle >= 0)<BR>+
{<BR>+
*ia_handle =
(DAT_IA_HANDLE)return_handle;<BR>+
}<BR>+<BR>+ if ( dat_ia_query(
*ia_handle,<BR>+
NULL,<BR>+
0,<BR>+
NULL,<BR>+
DAT_PROVIDER_FIELD_PROVIDER_SPECIFIC_ATTR,<BR>+
&p_attr ) == DAT_SUCCESS
)<BR>+
{<BR>+ for
( i = 0; i < p_attr.num_provider_specific_attr; i++
)<BR>+
{<BR>+
if ( (strcmp(
p_attr.provider_specific_attr[i].name,<BR>+
"DAT_EXTENSION_INTERFACE"
) == 0)
&&<BR>+
(strcmp(
p_attr.provider_specific_attr[i].value,<BR>+
"TRUE" ) == 0)
)<BR>+
{<BR>+
dat_os_dbg_print(DAT_OS_DBG_TYPE_CONSUMER_API,<BR>+
"DAT Registry: dat_ia_open ()
"<BR>+
"DAPL Extension Interface
supported!\n");<BR>+<BR>+
dat_extensions =
1;<BR>+
break;<BR>+
}<BR>+
}<BR>+
}<BR> }<BR> <BR> return
dat_status;<BR>Index:
README<BR>===================================================================<BR>---
README (revision 4589)<BR>+++ README (working copy)<BR>@@ -1,5
+1,10 @@<BR> There are now 3 uDAPL providers for openib
(openib,openib_scm,openib_cma). <BR> <BR>+NEW FEATURES for openib_cma
provider:<BR>+API extensions for immediate data and atomic operations have
been added.<BR>+see dat/include/dat/dat_extensions.h for new API's. <BR>+see
dapl/test/dtest/dtest_ext.c for example test cast
<BR>+<BR> <BR> ==========<BR> 1.0
BUILD:<BR><o:p> </o:p></SPAN></FONT></P><PRE><FONT face="Courier New" size=2><SPAN style="FONT-SIZE: 10pt"><o:p> </o:p></SPAN></FONT></PRE>
<P class=MsoNormal><FONT face=Arial size=2><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: Arial"><o:p> </o:p></SPAN></FONT></P></DIV></BLOCKQUOTE><!-- |**|begin egp html banner|**| --><BR>
<DIV style="WIDTH: 500px; COLOR: #909090; TEXT-ALIGN: center">
<HR style="WIDTH: 500px; BORDER-BOTTOM: 1px; TEXT-ALIGN: left">
<TT>YAHOO! GROUPS LINKS</TT> </DIV><BR>
<UL><TT>
<LI type=square> Visit your group "<A
href="http://groups.yahoo.com/group/dat-discussions">dat-discussions</A>" on
the web.<BR> </TT> <TT>
<LI type=square> To unsubscribe from this group, send an email
to:<BR> <A
href="mailto:dat-discussions-unsubscribe@yahoogroups.com?subject=Unsubscribe">dat-discussions-unsubscribe@yahoogroups.com</A><BR> </TT>
<TT>
<LI type=square> Your use of Yahoo! Groups is subject to the <A
href="http://docs.yahoo.com/info/terms/">Yahoo! Terms of Service</A>.</TT>
</LI></UL><BR>
<DIV style="WIDTH: 500px; COLOR: #909090; TEXT-ALIGN: center">
<HR style="WIDTH: 500px; BORDER-BOTTOM: 1px; TEXT-ALIGN: left">
</DIV><BR></BLOCKQUOTE><!-- |**|end egp html banner|**| --></BODY></HTML>