[openib-general] RE: [dat-discussions] RE: [RFC][PATCH] OpenIB uDAPL extension proposal - sample immed data and atomic api's
Kanevsky, Arkady
Arkady.Kanevsky at netapp.com
Fri Dec 23 12:07:45 PST 2005
Actually,
iWARP does not support neither immediate data nor atomic ops.
Arkady
Arkady Kanevsky email: arkady at netapp.com
Network Appliance Inc. phone: 781-768-5395
1601 Trapelo Rd. - Suite 16. Fax: 781-895-1195
Waltham, MA 02451 central phone: 781-768-5300
________________________________
From: Kanevsky, Arkady
Sent: Friday, December 23, 2005 2:55 PM
To: Arlin Davis; Lentini, James
Cc: openib-general at openib.org; dat-discussions at yahoogroups.com
Subject: [dat-discussions] RE: [RFC][PATCH] OpenIB uDAPL
extension proposal - sample immed data and atomic api's
Arlin,
nice proposal, thanks.
I have one high level question and a few specific technical
ones.
1. Why do you want to provide this functionality via extension
instead of part of new DAT spec, say 2.0?
This will allow Consumers to use all events, operations, and
Provider/IA functionality uniformly instead
of via 2 separate layers. This will also ensure that this basic
funcionality can be provided by all DAPL Provider
the same way on DAPL and DAT layers.
DAPL 2.0 is not done yet so we have time to incorporate that.
DAPL 2.0 already introduced new functionality which is easy to
beef up for your proposal.
See DAT_DTOS for example. DAT_EVENT is also modified to handle
remote invalidation so
a small addition for Immediate data and Atoimc ops is a sensible
addition.
This should simplify proposal significantly. As you will not
need to introduce any new
EXT structures.
In general, extension route was intended for RNIC|HCA providers
to expose HW capabilities beyond
IBTA, iWARP and VIA standards. The standard RDMA functionality
is best handle via spec addition.
DAT 2.0 does it for FMR, remote and local memory invalidation as
well as others.
I had posted a complete list of changes/addition to DAT 2.0
about a month ago.
But we had not discussed yet version change from 1.3 to 2.0 nor
how much backwards compatibility spec
will provide.
2. What is IMMED_EVENT? is it just immediate data without any
payload one?
I suggest chnaging the name so it will not use "EVENT". Just
call it NO_PAYLOAD.
Do you want to support 2 different way to delivery immediate
data?
One in event and one in data payload?
Why? I would think that just an event way will do.
3. I suggest beefing up DAT_DTO_COMPLETION_EVENT_DATA and
DAT_DTOS
to convey which operation completed and return Immediate data if
complete operation had immediate data.
Since we already modified these 2 struct as part of DAT 2.0
change lets add your proposal to the change.
This will allow Consumers to use single approach to deal with
completions, extension to the current one
but not a structural one. No need for DAT_EXTENSION_DATA,
DAT_EXT_EVENT_TYPE, DAT_EXT_OP
nor the whole mechanism for extended ops.
4. What is the purpose of DAT_EXT_WRITE_CONFIRM_FLAG? Is it to
expose IB round trip semantic?
iWARP does not support immediate data. One can try to format
payload to pass immediate data.
Is that what you had in mind?
What is the semantic meaning of the completion with this flag
set? without flag set?
Are extended flags are additonal values for COMPLETION_FLAGS?
2.4.1 talks about extended flags
but where they are passed in is not defined.
DAT 2.0 extended them already for FMR barrier. I would prefer to
follow that route rather than creating a separate
extension completion flags.
5. Why do you need RECV_IMMED? If Immed data is delivered in
event no new Recv operation is needed.
If Consumer asks for immediate data in payload where in payload
will it be?
If this is needed for local match for remote RDMA_Write to
handle immediate data lets state so.
What happens for mismatch between local and remote op? That is
recv was posted for Send and RDMA_Write
"arrived"? Vice Versa?
6. I see extension for immediate data for rdma_write but not for
send. Is this deliberate? If we are going
to extend DAT semantic to support Immediate data we can as well
support the full IBTA/iWARP functionality for it.
7. Currently memory registration do not support access to LMR or
RMR by Atomic ops.
Do you propose to extend the meaning of current MEM_PRIV for LMR
and RMR to covers atomic accesses
or add new values to LMR_MEM_PRIV and RMR_MEM_PRIV for atomic
operation support?
8. Any alignment requirements for memory used for atomic ops?
9. Any correlation requirements for SRQ buffers to support recv
with immediate data?
Have a great holidays,
Arkady
Arkady Kanevsky email: arkady at netapp.com
Network Appliance Inc. phone: 781-768-5395
1601 Trapelo Rd. - Suite 16. Fax: 781-895-1195
Waltham, MA 02451 central phone: 781-768-5300
________________________________
From: Arlin Davis [mailto:arlin.r.davis at intel.com]
Sent: Thursday, December 22, 2005 6:20 PM
To: Lentini, James; Kanevsky, Arkady
Cc: openib-general at openib.org;
dat-discussions at yahoogroups.com
Subject: [RFC][PATCH] OpenIB uDAPL extension proposal -
sample immed data and atomic api's
James and Arkady,
DAPL provides a generalized abstraction to RDMA capable
transports. As a generalized abstraction, it cannot exploit the unique
properties that many of the underlying platforms/interconnects can
provide so I would like to propose a simple (minimum impact on libdat)
extensible interface to uDAPL that will allow vendors to expose such
capabilities. I am looking for feedback, especially from the DAT
collaborative. I have included both a design document and actual working
code as a reference.
The patch provides a fully tested DAT and DAPL library
(openib_cma) set with the following provider extensions:
DAT_RETURN
dat_ep_post_write_immed(
IN DAT_EP_HANDLE ep_handle,
IN DAT_COUNT num_segments
IN DAT_LMR_TRIPLET *local_iov,
IN DAT_DTO_COOKIE user_cookie,
IN DAT_RMR_TRIPLE *remote_iov,
IN DAT_UINT32 immediate_data,
IN DAT_COMPLETION_FLAGS completion_flags);
DAT_RETURN
dat_ep_post_cmp_and_swap(
IN DAT_EP_HANDLE ep_handle,
IN DAT_UINT64 cmp_value,
IN DAT_UINT64 swap_value,
IN DAT_LMR_TRIPLE *local_iov,
IN DAT_DTO_COOKIE user_cookie,
IN DAT_RMR_TRIPLE *remote_iov,
IN DAT_COMPLETION_FLAGS completion_flags);
DAT_RETURN
dat_ep_post_fetch_and_add(
IN DAT_EP_HANDLE ep_handle,
IN DAT_UINT64 add_value,
IN DAT_LMR_TRIPLE *local_iov,
IN DAT_DTO_COOKIE user_cookie,
IN DAT_RMR_TRIPLE *remote_iov,
IN DAT_COMPLETION_FLAGS completion_flags);
Also, included is a sample program (dtest_ext.c) that
can be used as a programming example.
Thanks,
-arlin
Signed-off by: Arlin Davis <ardavis at ichips.intel.com>
<mailto:ardavis at ichips.intel.com>
Index: test/dtest/dat.conf
===================================================================
--- test/dtest/dat.conf (revision 4589)
+++ test/dtest/dat.conf (working copy)
@@ -1,11 +1,20 @@
#
-# DAT 1.1 and 1.2 configuration file
+# DAT 1.2 configuration file
#
# Each entry should have the following fields:
#
# <ia_name> <api_version> <threadsafety> <default>
<lib_path> \
# <provider_version> <ia_params>
<platform_params>
#
-# Example for openib using the first Mellanox adapter,
port 1 and port 2
+# Example for openib_cma and openib_scm
+#
+# For scm version you specify <ia_params> as actual
device name and port
+# For cma version you specify <ia_params> as:
+# network address, network hostname, or netdev
name and 0 for port
+#
+OpenIB-scm1 u1.2 nonthreadsafe default
/usr/local/openib_dapl/udapl/Target/libdapl.so mv_dapl.1.2 "mthca0 1" ""
+OpenIB-scm2 u1.2 nonthreadsafe default
/usr/local/openib_dapl/udapl/Target/libdapl.so mv_dapl.1.2 "mthca0 2" ""
+OpenIB-cma-ip u1.2 nonthreadsafe default
/usr/local/openib_dapl/udapl/Target/libdapl.so mv_dapl.1.2 "192.168.0.22
0" ""
+OpenIB-cma-name u1.2 nonthreadsafe default
/usr/local/openib_dapl/udapl/Target/libdapl.so mv_dapl.1.2 "svr1-ib0 0"
""
+OpenIB-cma-netdev u1.2 nonthreadsafe default
/usr/local/openib_dapl/udapl/Target/libdapl.so mv_dapl.1.2 "ib0 0" ""
-IB1 u1.2 nonthreadsafe default
Index: test/dtest/makefile
===================================================================
--- test/dtest/makefile (revision 4589)
+++ test/dtest/makefile (working copy)
@@ -4,13 +4,18 @@ CFLAGS = -O2 -g
DAT_INC = ../../dat/include
DAT_LIB = /usr/local/lib
-all: dtest
+all: dtest dtest_ext
clean:
- rm -f *.o;touch *.c;rm -f dtest
+ rm -f *.o;touch *.c;rm -f dtest dtest_ext
dtest: ./dtest.c
$(CC) $(CFLAGS) ./dtest.c -o dtest \
-DDAPL_PROVIDER='"OpenIB-cma-ip"' \
-I $(DAT_INC) -L $(DAT_LIB) -ldat
+dtest_ext: ./dtest_ext.c
+ $(CC) $(CFLAGS) ./dtest_ext.c -o dtest_ext \
+ -DDAPL_PROVIDER='"OpenIB-cma-ip"' \
+ -I $(DAT_INC) -L $(DAT_LIB) -ldat
+
Index: test/dtest/README
===================================================================
--- test/dtest/README (revision 4589)
+++ test/dtest/README (working copy)
@@ -1,10 +1,11 @@
simple dapl test just for initial openIB uDAPL
testing...
dtest/dtest.c
+ dtest/dtest_ext.c
dtest/makefile
dtest/dat.conf
-to build (default uDAPL name == IB1, ib device ==
mthca0, port == 1)
+to build (default uDAPL name == OpenIB-cma-ip)
edit makefile and change path (DAT_LIB) to
appropriate libdat.so
edit dat.conf and change path to appropriate
libdapl.so
cp dat.conf to /etc/dat.conf
Index: dapl/include/dapl.h
===================================================================
--- dapl/include/dapl.h (revision 4589)
+++ dapl/include/dapl.h (working copy)
@@ -1,25 +1,28 @@
/*
- * Copyright (c) 2002-2003, Network Appliance, Inc. All
rights reserved.
+ * Copyright (c) 2002-2005, Network Appliance, Inc. All
rights reserved.
*
* This Software is licensed under one of the following
licenses:
- *
+ *
* 1) under the terms of the "Common Public License
1.0" a copy of which is
- * available from the Open Source Initiative, see
+ * in the file LICENSE.txt in the root directory.
The license is also
+ * available from the Open Source Initiative, see
* http://www.opensource.org/licenses/cpl.php.
- *
- * 2) under the terms of the "The BSD License" a copy
of which is
- * available from the Open Source Initiative, see
+ *
+ * 2) under the terms of the "The BSD License" a copy
of which is in the file
+ * LICENSE2.txt in the root directory. The license
is also available from
+ * the Open Source Initiative, see
*
http://www.opensource.org/licenses/bsd-license.php.
- *
+ *
* 3) under the terms of the "GNU General Public
License (GPL) Version 2" a
- * copy of which is available from the Open Source
Initiative, see
+ * copy of which is in the file LICENSE3.txt in the
root directory. The
+ * license is also available from the Open Source
Initiative, see
*
http://www.opensource.org/licenses/gpl-license.php.
- *
+ *
* Licensee has the right to choose one of the above
licenses.
- *
+ *
* Redistributions of source code must retain the above
copyright
* notice and one of the license notices.
- *
+ *
* Redistributions in binary form must reproduce both
the above copyright
* notice, one of the license notices in the
documentation
* and/or other materials provided with the
distribution.
@@ -61,6 +64,8 @@
#include "dapl_dummy_util.h"
#elif OPENIB
#include "dapl_ib_util.h"
+#elif DET
+#include "dapl_det_util.h"
#endif
/*********************************************************************
@@ -213,6 +218,10 @@ typedef struct dapl_cookie
DAPL_COOKIE;
typedef struct dapl_dto_cookie
DAPL_DTO_COOKIE;
typedef struct dapl_rmr_cookie
DAPL_RMR_COOKIE;
+#ifdef DAPL_EXTENSIONS
+typedef struct dapl_ext_cookie
DAPL_EXT_COOKIE;
+#endif
+
typedef struct dapl_private DAPL_PRIVATE;
typedef void (*DAPL_CONNECTION_STATE_HANDLER) (
@@ -563,6 +572,13 @@ typedef enum dapl_dto_type
DAPL_DTO_TYPE_RECV,
DAPL_DTO_TYPE_RDMA_WRITE,
DAPL_DTO_TYPE_RDMA_READ,
+#ifdef DAPL_EXTENSIONS
+ DAPL_DTO_TYPE_RDMA_WRITE_IMMED,
+ DAPL_DTO_TYPE_RECV_IMMED,
+ DAPL_DTO_TYPE_CMP_AND_SWAP,
+ DAPL_DTO_TYPE_FETCH_AND_ADD,
+#endif
+
} DAPL_DTO_TYPE;
typedef enum dapl_cookie_type
@@ -570,6 +586,9 @@ typedef enum dapl_cookie_type
DAPL_COOKIE_TYPE_NULL,
DAPL_COOKIE_TYPE_DTO,
DAPL_COOKIE_TYPE_RMR,
+#ifdef DAPL_EXTENSIONS
+ DAPL_COOKIE_TYPE_EXTENSION,
+#endif
} DAPL_COOKIE_TYPE;
/* DAPL_DTO_COOKIE used as context for DTO WQEs */
@@ -587,6 +606,27 @@ struct dapl_rmr_cookie
DAT_RMR_COOKIE cookie;
};
+#ifdef DAPL_EXTENSIONS
+
+/* DAPL extended cookie types */
+typedef enum dapl_ext_type
+{
+ DAPL_EXT_TYPE_RDMA_WRITE_IMMED,
+ DAPL_EXT_TYPE_CMP_AND_SWAP,
+ DAPL_EXT_TYPE_FETCH_AND_ADD,
+ DAPL_EXT_TYPE_RECV
+} DAPL_EXT_TYPE;
+
+/* DAPL extended cookie */
+struct dapl_ext_cookie
+{
+ DAPL_EXT_TYPE type;
+ DAT_DTO_COOKIE cookie;
+ DAT_COUNT size; /* used RDMA write
with immed */
+};
+
+#endif
+
/* DAPL_COOKIE used as context for WQEs */
struct dapl_cookie
{
@@ -597,6 +637,9 @@ struct dapl_cookie
{
DAPL_DTO_COOKIE dto;
DAPL_RMR_COOKIE rmr;
+#ifdef DAPL_EXTENSIONS
+ DAPL_EXT_COOKIE ext;
+#endif
} val;
};
@@ -1116,6 +1159,15 @@ extern DAT_RETURN
dapl_srq_set_lw(
IN DAT_SRQ_HANDLE, /* srq_handle
*/
IN DAT_COUNT); /* low_watermark
*/
+#ifdef DAPL_EXTENSIONS
+
+extern DAT_RETURN dapl_extensions(
+ IN DAT_HANDLE, /* dat_handle
*/
+ IN DAT_EXT_OP, /* extension
operation */
+ IN va_list ); /* va_list args */
+
+#endif
+
/*
* DAPL internal utility function prototpyes
*/
Index: dapl/udapl/Makefile
===================================================================
--- dapl/udapl/Makefile (revision 4589)
+++ dapl/udapl/Makefile (working copy)
@@ -156,6 +156,7 @@ PROVIDER = $(TOPDIR)/../openib_cma
CFLAGS += -DOPENIB
CFLAGS += -DCQ_WAIT_OBJECT
CFLAGS += -I/usr/local/include/infiniband
+CFLAGS += -I/usr/local/include/rdma
endif
#
@@ -168,6 +169,12 @@ endif
# VN_MEM_SHARED_VIRTUAL_SUPPORT
# CFLAGS += -DVN_MEM_SHARED_VIRTUAL_SUPPORT=1
+# If an implementation supports DAPL extensions
+CFLAGS += -DDAPL_EXTENSIONS
+
+# If an implementation supports DAPL provider specific
attributes
+CFLAGS += -DDAPL_PROVIDER_SPECIFIC_ATTR
+
CFLAGS += -I.
CFLAGS += -I..
CFLAGS += -I../../dat/include
@@ -283,6 +290,8 @@ LDFLAGS += -libverbs -lrdmacm
LDFLAGS += -rpath /usr/local/lib -L /usr/local/lib
PROVIDER_SRCS = dapl_ib_util.c dapl_ib_cq.c
dapl_ib_qp.c \
dapl_ib_cm.c dapl_ib_mem.c
+# implementation supports DAPL extensions
+PROVIDER_SRCS += dapl_ib_extensions.c
endif
UDAPL_SRCS = dapl_init.c \
Index: dapl/common/dapl_ia_query.c
===================================================================
--- dapl/common/dapl_ia_query.c (revision 4589)
+++ dapl/common/dapl_ia_query.c (working copy)
@@ -167,6 +167,14 @@ dapl_ia_query (
#if !defined(__KDAPL__)
provider_attr->pz_support =
DAT_PZ_UNIQUE;
#endif /* !KDAPL */
+
+ /*
+ * Have provider set their own.
+ */
+#ifdef DAPL_PROVIDER_SPECIFIC_ATTR
+ dapls_set_provider_specific_attr(provider_attr);
+#endif
+
/*
* Set up evd_stream_merging_supported options.
Note there is
* one bit per allowable combination, using the
ordinal
Index: dapl/common/dapl_adapter_util.h
===================================================================
--- dapl/common/dapl_adapter_util.h (revision 4589)
+++ dapl/common/dapl_adapter_util.h (working copy)
@@ -256,6 +256,21 @@ dapls_ib_wait_object_wait (
IN u_int32_t timeout);
#endif
+#ifdef DAPL_PROVIDER_SPECIFIC_ATTR
+void
+dapls_set_provider_specific_attr(
+ IN DAT_PROVIDER_ATTR *provider_attr );
+#endif
+
+#ifdef DAPL_EXTENSIONS
+void
+dapls_cqe_to_event_extension(
+ IN DAPL_EP *ep_ptr,
+ IN DAPL_COOKIE *cookie,
+ IN ib_work_completion_t *cqe_ptr,
+ OUT DAT_EVENT *event_ptr);
+#endif
+
/*
* Values for provider DAT_NAMED_ATTR
*/
@@ -272,6 +287,8 @@ dapls_ib_wait_object_wait (
#include "dapl_dummy_dto.h"
#elif OPENIB
#include "dapl_ib_dto.h"
+#elif DET
+#include "dapl_det_dto.h"
#endif
Index: dapl/common/dapl_provider.c
===================================================================
--- dapl/common/dapl_provider.c (revision 4589)
+++ dapl/common/dapl_provider.c (working copy)
@@ -221,7 +221,11 @@ DAT_PROVIDER
g_dapl_provider_template =
&dapl_srq_post_recv,
&dapl_srq_query,
&dapl_srq_resize,
- &dapl_srq_set_lw
+ &dapl_srq_set_lw,
+
+#ifdef DAPL_EXTENSIONS
+ &dapl_extensions
+#endif
};
#endif /* __KDAPL__ */
Index: dapl/common/dapl_evd_util.c
===================================================================
--- dapl/common/dapl_evd_util.c (revision 4589)
+++ dapl/common/dapl_evd_util.c (working copy)
@@ -502,6 +502,20 @@ dapli_evd_eh_print_cqe (
#ifdef DAPL_DBG
static char *optable[] =
{
+#ifdef OPENIB
+ /* different order for openib verbs */
+ "OP_RDMA_WRITE",
+ "OP_RDMA_WRITE_IMM",
+ "OP_SEND",
+ "OP_SEND_IMM",
+ "OP_RDMA_READ",
+ "OP_COMP_AND_SWAP",
+ "OP_FETCH_AND_ADD",
+ "OP_RECEIVE",
+ "OP_RECEIVE_IMM",
+ "OP_BIND_MW",
+ "OP_INVALID",
+#else
"OP_SEND",
"OP_RDMA_READ",
"OP_RDMA_WRITE",
@@ -509,6 +523,7 @@ dapli_evd_eh_print_cqe (
"OP_FETCH_AND_ADD",
"OP_RECEIVE",
"OP_BIND_MW",
+#endif
0
};
@@ -1113,6 +1128,15 @@ dapli_evd_cqe_to_event (
dapls_cookie_dealloc (&ep_ptr->req_buffer,
cookie);
break;
}
+
+#ifdef DAPL_EXTENSIONS
+ case DAPL_COOKIE_TYPE_EXTENSION:
+ {
+ dapls_cqe_to_event_extension(ep_ptr, cookie,
cqe_ptr, event_ptr);
+ break;
+ }
+#endif
+
default:
{
dapl_os_assert (!"Invalid Operation type");
Index: dapl/openib_cma/dapl_ib_dto.h
===================================================================
--- dapl/openib_cma/dapl_ib_dto.h (revision 4589)
+++ dapl/openib_cma/dapl_ib_dto.h (working copy)
@@ -35,7 +35,7 @@
*
* Description:
*
- * The uDAPL openib provider - DTO operations and CQE
macros
+ * The OpenIB uCMA provider - DTO operations and CQE
macros
*
************************************************************************
****
* Source Control System Information
@@ -119,7 +119,6 @@ dapls_ib_post_recv (
return DAT_SUCCESS;
}
-
/*
* dapls_ib_post_send
*
@@ -191,7 +190,7 @@ dapls_ib_post_send (
if (cookie != NULL)
cookie->val.dto.size = total_len;
-
+
if ((op_type == OP_RDMA_WRITE) || (op_type ==
OP_RDMA_READ)) {
wr.wr.rdma.remote_addr =
remote_iov->target_address;
wr.wr.rdma.rkey = remote_iov->rmr_context;
@@ -224,6 +223,152 @@ dapls_ib_post_send (
return DAT_SUCCESS;
}
+#ifdef DAPL_EXTENSIONS
+/*
+ * dapls_ib_post_ext_send
+ *
+ * Provider specific extended Post SEND function
+ */
+STATIC _INLINE_ DAT_RETURN
+dapls_ib_post_ext_send (
+ IN DAPL_EP *ep_ptr,
+ IN ib_send_op_type_t op_type,
+ IN DAPL_COOKIE *cookie,
+ IN DAT_COUNT segments,
+ IN DAT_LMR_TRIPLET *local_iov,
+ IN const DAT_RMR_TRIPLET *remote_iov,
+ IN DAT_UINT32 idata,
+ IN DAT_UINT64 compare_add,
+ IN DAT_UINT64 swap,
+ IN DAT_COMPLETION_FLAGS completion_flags)
+{
+ dapl_dbg_log(DAPL_DBG_TYPE_EP,
+ " post_snd: ep %p op %d ck %p sgs",
+ "%d l_iov %p r_iov %p f %d\n",
+ ep_ptr, op_type, cookie, segments,
local_iov,
+ remote_iov, completion_flags);
+
+ ib_data_segment_t ds_array[DEFAULT_DS_ENTRIES];
+ ib_data_segment_t *ds_array_p;
+ struct ibv_send_wr wr;
+ struct ibv_send_wr *bad_wr;
+ ib_hca_transport_t *ibt_ptr =
+ &ep_ptr->header.owner_ia->hca_ptr->ib_trans;
+ DAT_COUNT i, total_len;
+
+ dapl_dbg_log(DAPL_DBG_TYPE_EP,
+ " post_snd: ep %p cookie %p segs %d
l_iov %p\n",
+ ep_ptr, cookie, segments, local_iov);
+
+ if(segments <= DEFAULT_DS_ENTRIES)
+ ds_array_p = ds_array;
+ else
+ ds_array_p =
+ dapl_os_alloc(segments *
sizeof(ib_data_segment_t));
+
+ if (NULL == ds_array_p)
+ return (DAT_INSUFFICIENT_RESOURCES);
+
+ /* setup the work request */
+ wr.next = 0;
+ wr.opcode = op_type;
+ wr.num_sge = 0;
+ wr.send_flags = 0;
+ wr.wr_id = (uint64_t)(uintptr_t)cookie;
+ wr.sg_list = ds_array_p;
+ total_len = 0;
+
+ for (i = 0; i < segments; i++ ) {
+ if ( !local_iov[i].segment_length )
+ continue;
+
+ ds_array_p->addr = (uint64_t)
local_iov[i].virtual_address;
+ ds_array_p->length =
local_iov[i].segment_length;
+ ds_array_p->lkey = local_iov[i].lmr_context;
+
+ dapl_dbg_log(DAPL_DBG_TYPE_EP,
+ " post_snd: lkey 0x%x va %p len
%d\n",
+ ds_array_p->lkey,
ds_array_p->addr,
+ ds_array_p->length );
+
+ total_len += ds_array_p->length;
+ wr.num_sge++;
+ ds_array_p++;
+ }
+
+ if (cookie != NULL)
+ cookie->val.dto.size = total_len;
+
+ if ((op_type == OP_RDMA_WRITE) ||
+ (op_type == OP_RDMA_WRITE_IMM) ||
+ (op_type == OP_RDMA_READ)) {
+ wr.wr.rdma.remote_addr =
remote_iov->target_address;
+ wr.wr.rdma.rkey = remote_iov->rmr_context;
+ dapl_dbg_log(DAPL_DBG_TYPE_EP,
+ " post_snd_rdma: rkey 0x%x va
%#016Lx\n",
+ wr.wr.rdma.rkey,
wr.wr.rdma.remote_addr);
+ }
+
+ switch (op_type) {
+ case OP_RDMA_WRITE_IMM:
+ dapl_dbg_log(DAPL_DBG_TYPE_EP,
+ " post_snd:
OP_RDMA_WRITE_IMMED=0x%x\n", idata );
+ wr.imm_data = idata;
+ break;
+ case OP_COMP_AND_SWAP:
+ /* OP_COMP_AND_SWAP has direct IBAL wr_type
mapping */
+ dapl_dbg_log(DAPL_DBG_TYPE_EP,
+ " post_snd:
OP_COMP_AND_SWAP=%lx,"
+ "%lx rkey 0x%x va %#016Lx\n",
+ compare_add, swap,
remote_iov->rmr_context,
+ remote_iov->target_address);
+
+ wr.wr.atomic.compare_add = compare_add;
+ wr.wr.atomic.swap = swap;
+ wr.wr.atomic.remote_addr =
remote_iov->target_address;
+ wr.wr.atomic.rkey = remote_iov->rmr_context;
+ break;
+ case OP_FETCH_AND_ADD:
+ /* OP_FETCH_AND_ADD has direct IBAL wr_type
mapping */
+ dapl_dbg_log(DAPL_DBG_TYPE_EP,
+ " post_snd:
OP_FETCH_AND_ADD=%lx,"
+ "%lx rkey 0x%x va %#016Lx\n",
+ compare_add,
remote_iov->rmr_context,
+ remote_iov->target_address);
+
+ wr.wr.atomic.compare_add = compare_add;
+ wr.wr.atomic.remote_addr =
remote_iov->target_address;
+ wr.wr.atomic.rkey = remote_iov->rmr_context;
+ break;
+ default:
+ break;
+ }
+
+ /* inline data for send or write ops */
+ if ((total_len <= ibt_ptr->max_inline_send) &&
+ ((op_type == OP_SEND) || (op_type ==
OP_RDMA_WRITE)))
+ wr.send_flags |= IBV_SEND_INLINE;
+
+ /* set completion flags in work request */
+ wr.send_flags |= (DAT_COMPLETION_SUPPRESS_FLAG &
+ completion_flags) ? 0 :
IBV_SEND_SIGNALED;
+ wr.send_flags |=
(DAT_COMPLETION_BARRIER_FENCE_FLAG &
+ completion_flags) ?
IBV_SEND_FENCE : 0;
+ wr.send_flags |=
(DAT_COMPLETION_SOLICITED_WAIT_FLAG &
+ completion_flags) ?
IBV_SEND_SOLICITED : 0;
+
+ dapl_dbg_log(DAPL_DBG_TYPE_EP,
+ " post_snd: op 0x%x flags 0x%x sglist
%p, %d\n",
+ wr.opcode, wr.send_flags, wr.sg_list,
wr.num_sge);
+
+ if (ibv_post_send(ep_ptr->qp_handle->cm_id->qp,
&wr, &bad_wr))
+ return(
dapl_convert_errno(EFAULT,"ibv_recv") );
+
+ dapl_dbg_log(DAPL_DBG_TYPE_EP," post_snd:
returned\n");
+ return DAT_SUCCESS;
+}
+#endif
+
STATIC _INLINE_ DAT_RETURN
dapls_ib_optional_prv_dat(
IN DAPL_CR *cr_ptr,
Index: dapl/openib_cma/dapl_ib_util.c
===================================================================
--- dapl/openib_cma/dapl_ib_util.c (revision 4589)
+++ dapl/openib_cma/dapl_ib_util.c (working copy)
@@ -35,7 +35,7 @@
*
* Description:
*
- * The uDAPL openib provider - init, open, close,
utilities, work thread
+ * The OpenIB uCMA provider - init, open, close,
utilities, work thread
*
************************************************************************
****
* Source Control System Information
@@ -64,7 +64,6 @@ static const char rcsid[] = "$Id: $";
#include <net/if.h> /* for struct ifreq */
#include <net/if_arp.h> /* for ARPHRD_INFINIBAND */
-
int g_dapl_loopback_connection = 0;
int g_ib_pipe[2];
ib_thread_state_t g_ib_thread_state = 0;
@@ -727,7 +726,7 @@ void dapli_thread(void *arg)
int ret,idx,fds;
char rbuf[2];
- dapl_dbg_log (DAPL_DBG_TYPE_CM,
+ dapl_dbg_log (DAPL_DBG_TYPE_UTIL,
" ib_thread(%d,0x%x): ENTER: pipe %d
ucma %d\n",
getpid(), g_ib_thread, g_ib_pipe[0],
rdma_get_fd());
@@ -767,7 +766,7 @@ void dapli_thread(void *arg)
ufds[idx].revents = 0;
uhca[idx] = hca;
- dapl_dbg_log(DAPL_DBG_TYPE_CM,
+ dapl_dbg_log(DAPL_DBG_TYPE_UTIL,
" ib_thread(%d) poll_fd:
hca[%d]=%p, async=%d"
" pipe=%d cm=%d cq=d\n",
getpid(), hca, ufds[idx-1].fd,
@@ -783,14 +782,14 @@ void dapli_thread(void *arg)
dapl_os_unlock(&g_hca_lock);
ret = poll(ufds, fds, -1);
if (ret <= 0) {
- dapl_dbg_log(DAPL_DBG_TYPE_WARN,
+ dapl_dbg_log(DAPL_DBG_TYPE_UTIL,
" ib_thread(%d): ERR %s
poll\n",
getpid(),strerror(errno));
dapl_os_lock(&g_hca_lock);
continue;
}
- dapl_dbg_log(DAPL_DBG_TYPE_CM,
+ dapl_dbg_log(DAPL_DBG_TYPE_UTIL,
" ib_thread(%d) poll_event: "
" async=0x%x pipe=0x%x cm=0x%x
cq=0x%x\n",
getpid(), ufds[idx-1].revents,
ufds[0].revents,
@@ -834,3 +833,63 @@ void dapli_thread(void *arg)
dapl_os_unlock(&g_hca_lock);
}
+#ifdef DAPL_PROVIDER_SPECIFIC_ATTR
+/*
+ * dapls_set_provider_specific_attr
+ *
+ * Input:
+ * attr_ptr Pointer provider attributes
+ *
+ * Output:
+ * none
+ *
+ * Returns:
+ * void
+ */
+DAT_NAMED_ATTR ib_attrs[] = {
+
+#ifdef DAPL_EXTENSIONS
+ {
+ DAT_EXT_ATTR,
+ DAT_EXT_ATTR_TRUE
+ },
+ {
+ DAT_EXT_ATTR_RDMA_WRITE_IMMED,
+ DAT_EXT_ATTR_TRUE
+ },
+ {
+ DAT_EXT_ATTR_RECV_IMMED,
+ DAT_EXT_ATTR_TRUE
+ },
+ /* inbound immediate data placed in event, NOT
payload */
+ {
+ DAT_EXT_ATTR_RECV_IMMED_EVENT,
+ DAT_EXT_ATTR_TRUE
+ },
+ {
+ DAT_EXT_ATTR_FETCH_AND_ADD,
+ DAT_EXT_ATTR_TRUE
+ },
+ {
+ DAT_EXT_ATTR_CMP_AND_SWAP,
+ DAT_EXT_ATTR_TRUE
+ },
+#else
+ {
+ "DAT_EXTENSION_INTERFACE",
+ "FALSE"
+ },
+#endif
+};
+
+#define SPEC_ATTR_SIZE(x) (
sizeof(x)/sizeof(DAT_NAMED_ATTR) )
+
+void dapls_set_provider_specific_attr(
+ IN DAT_PROVIDER_ATTR *attr_ptr )
+{
+ attr_ptr->num_provider_specific_attr =
SPEC_ATTR_SIZE(ib_attrs);
+ attr_ptr->provider_specific_attr = ib_attrs;
+}
+
+#endif
+
Index: dapl/openib_cma/dapl_ib_mem.c
===================================================================
--- dapl/openib_cma/dapl_ib_mem.c (revision 4589)
+++ dapl/openib_cma/dapl_ib_mem.c (working copy)
@@ -25,9 +25,9 @@
/**********************************************************************
*
- * MODULE: dapl_det_mem.c
+ * MODULE: dapl_ib_mem.c
*
- * PURPOSE: Intel DET APIs: Memory windows,
registration,
+ * PURPOSE: OpenIB uCMA provider Memory windows,
registration,
* and protection domain
*
* $Id: $
@@ -72,12 +72,10 @@ dapls_convert_privileges(IN
DAT_MEM_PRIV
access |= IBV_ACCESS_LOCAL_WRITE;
if (DAT_MEM_PRIV_REMOTE_WRITE_FLAG & privileges)
access |= IBV_ACCESS_REMOTE_WRITE;
- if (DAT_MEM_PRIV_REMOTE_READ_FLAG & privileges)
- access |= IBV_ACCESS_REMOTE_READ;
- if (DAT_MEM_PRIV_REMOTE_READ_FLAG & privileges)
- access |= IBV_ACCESS_REMOTE_READ;
- if (DAT_MEM_PRIV_REMOTE_READ_FLAG & privileges)
+ if (DAT_MEM_PRIV_REMOTE_READ_FLAG & privileges) {
access |= IBV_ACCESS_REMOTE_READ;
+ access |= IBV_ACCESS_REMOTE_ATOMIC;
+ }
return access;
}
Index: dapl/openib_cma/dapl_ib_cm.c
===================================================================
--- dapl/openib_cma/dapl_ib_cm.c (revision 4589)
+++ dapl/openib_cma/dapl_ib_cm.c (working copy)
@@ -35,7 +35,7 @@
*
* Description:
*
- * The uDAPL openib provider - connection management
+ * The OpenIB uCMA provider - uCMA connection
management
*
************************************************************************
****
* Source Control System Information
@@ -592,7 +592,11 @@ dapls_ib_setup_conn_listener(IN
DAPL_IA
if (rdma_bind_addr(conn->cm_id,
(struct sockaddr
*)&ia_ptr->hca_ptr->hca_address)) {
- dat_status =
dapl_convert_errno(errno,"setup_listener");
+ if (errno == -EBUSY)
+ dat_status = DAT_CONN_QUAL_IN_USE;
+ else
+ dat_status =
+
dapl_convert_errno(errno,"setup_listener");
goto bail;
}
Index: dapl/openib_cma/dapl_ib_qp.c
===================================================================
--- dapl/openib_cma/dapl_ib_qp.c (revision 4589)
+++ dapl/openib_cma/dapl_ib_qp.c (working copy)
@@ -25,9 +25,9 @@
/**********************************************************************
*
- * MODULE: dapl_det_qp.c
+ * MODULE: dapl_ib_qp.c
*
- * PURPOSE: QP routines for access to DET Verbs
+ * PURPOSE: OpenIB uCMA QP routines
*
* $Id: $
**********************************************************************/
Index: dapl/openib_cma/dapl_ib_util.h
===================================================================
--- dapl/openib_cma/dapl_ib_util.h (revision 4589)
+++ dapl/openib_cma/dapl_ib_util.h (working copy)
@@ -35,7 +35,7 @@
*
* Description:
*
- * The uDAPL openib provider - definitions,
prototypes,
+ * The OpenIB uCMA provider - definitions,
prototypes,
*
************************************************************************
****
* Source Control System Information
Index: dapl/openib_cma/README
===================================================================
--- dapl/openib_cma/README (revision 4589)
+++ dapl/openib_cma/README (working copy)
@@ -23,15 +23,22 @@ New files for openib_scm provider
dapl/openib_cma/dapl_ib_util.c
dapl/openib_cma/dapl_ib_util.h
dapl/openib_cma/dapl_ib_cm.c
+ dapl/openib_cma/dapl_ib_extensions.c
A simple dapl test just for openib_scm testing...
test/dtest/dtest.c
+ test/dtest/dtest_ext.c
test/dtest/makefile
server: dtest -s
client: dtest -h hostname
+or with extensions
+
+ server: dtest_ext -s
+ client: dtest_ext -h hostname
+
known issues:
no memory windows support in ibverbs,
dat_create_rmr fails.
Index: dapl/openib_cma/dapl_ib_cq.c
===================================================================
--- dapl/openib_cma/dapl_ib_cq.c (revision 4589)
+++ dapl/openib_cma/dapl_ib_cq.c (working copy)
@@ -35,7 +35,7 @@
*
* Description:
*
- * The uDAPL openib provider - completion queue
+ * The OpenIB uCMA provider - completion queue
*
************************************************************************
****
* Source Control System Information
@@ -498,7 +498,10 @@ dapls_ib_wait_object_wait(IN
ib_wait_obj
if (timeout != DAT_TIMEOUT_INFINITE)
timeout_ms = timeout/1000;
- status = poll(&cq_fd, 1, timeout_ms);
+ /* restart syscall */
+ while ((status = poll(&cq_fd, 1, timeout_ms)) ==
-1 )
+ if (errno == EINTR)
+ continue;
/* returned event */
if (status > 0) {
@@ -511,6 +514,8 @@ dapls_ib_wait_object_wait(IN
ib_wait_obj
/* timeout */
} else if (status == 0)
status = ETIMEDOUT;
+ else
+ status = errno;
dapl_dbg_log(DAPL_DBG_TYPE_UTIL,
" cq_object_wait: RET evd %p ibv_cq %p
ibv_ctx %p %s\n",
Index: dat/include/dat/dat_redirection.h
===================================================================
--- dat/include/dat/dat_redirection.h (revision
4589)
+++ dat/include/dat/dat_redirection.h (working copy)
@@ -59,10 +59,10 @@ typedef struct dat_provider
DAT_PROVIDER
* This would allow a good compiler to avoid
indirection overhead when
* making function calls.
*/
-
#define DAT_HANDLE_TO_PROVIDER(handle) (*(DAT_PROVIDER
**)(handle))
#endif
+
#define DAT_IA_QUERY(ia, evd, ia_msk, ia_ptr, p_msk,
p_ptr) \
(*DAT_HANDLE_TO_PROVIDER (ia)->ia_query_func) (\
(ia), \
@@ -395,6 +395,12 @@ typedef struct dat_provider
DAT_PROVIDER
(lbuf), \
(cookie))
+#define DAT_EXTENSION(handle, op, args) \
+ (*DAT_HANDLE_TO_PROVIDER (handle)->extension_func)
(\
+ (handle), \
+ (op), \
+ (args))
+
/***************************************************************
*
* FUNCTION PROTOTYPES
@@ -720,4 +726,11 @@ typedef DAT_RETURN
(*DAT_SRQ_POST_RECV_F
IN DAT_LMR_TRIPLET *, /* local_iov
*/
IN DAT_DTO_COOKIE ); /* user_cookie
*/
+/* Extension function */
+#include <stdarg.h>
+typedef DAT_RETURN (*DAT_EXTENSION_FUNC) (
+ IN DAT_HANDLE, /* dat handle */
+ IN DAT_EXT_OP, /* extension operation */
+ IN va_list ); /* va_list */
+
#endif /* _DAT_REDIRECTION_H_ */
Index: dat/include/dat/dat.h
===================================================================
--- dat/include/dat/dat.h (revision 4589)
+++ dat/include/dat/dat.h (working copy)
@@ -854,11 +854,15 @@ typedef enum dat_event_number
DAT_ASYNC_ERROR_EP_BROKEN =
0x08003,
DAT_ASYNC_ERROR_TIMED_OUT =
0x08004,
DAT_ASYNC_ERROR_PROVIDER_INTERNAL_ERROR =
0x08005,
- DAT_SOFTWARE_EVENT =
0x10001
+ DAT_SOFTWARE_EVENT =
0x10001,
+ DAT_EXTENSION_EVENT =
0x20001
+
} DAT_EVENT_NUMBER;
-/* Union for event Data */
+/* include extension data definitions */
+#include <dat/dat_extensions.h>
+/* Union for event Data */
typedef union dat_event_data
{
DAT_DTO_COMPLETION_EVENT_DATA
dto_completion_event_data;
@@ -867,6 +871,7 @@ typedef union dat_event_data
DAT_CONNECTION_EVENT_DATA
connect_event_data;
DAT_ASYNCH_ERROR_EVENT_DATA
asynch_error_event_data;
DAT_SOFTWARE_EVENT_DATA
software_event_data;
+ DAT_EXTENSION_DATA extension_data;
} DAT_EVENT_DATA;
/* Event struct that holds all event information */
@@ -1222,6 +1227,11 @@ extern DAT_RETURN dat_srq_set_lw
(
IN DAT_SRQ_HANDLE, /* srq_handle
*/
IN DAT_COUNT); /* low_watermark
*/
+extern DAT_RETURN dat_extension(
+ IN DAT_HANDLE,
+ IN DAT_EXT_OP,
+ IN ... );
+
/*
* DAT registry functions.
*
Index: dat/include/dat/udat_redirection.h
===================================================================
--- dat/include/dat/udat_redirection.h (revision
4589)
+++ dat/include/dat/udat_redirection.h (working copy)
@@ -199,7 +199,6 @@ typedef DAT_RETURN
(*DAT_EVD_SET_UNWAITA
typedef DAT_RETURN (*DAT_EVD_CLEAR_UNWAITABLE_FUNC) (
IN DAT_EVD_HANDLE); /* evd_handle
*/
-
#include <dat/dat_redirection.h>
struct dat_provider
@@ -294,6 +293,10 @@ struct dat_provider
DAT_SRQ_QUERY_FUNC srq_query_func;
DAT_SRQ_RESIZE_FUNC
srq_resize_func;
DAT_SRQ_SET_LW_FUNC
srq_set_lw_func;
+
+ /* extension for provder specific functions */
+ DAT_EXTENSION_FUNC extension_func;
+
};
#endif /* _UDAT_REDIRECTION_H_ */
Index: dat/include/dat/dat_extensions.h
===================================================================
--- dat/include/dat/dat_extensions.h (revision 0)
+++ dat/include/dat/dat_extensions.h (revision 0)
@@ -0,0 +1,209 @@
+/*
+ * Copyright (c) 2002-2005, Network Appliance, Inc. All
rights reserved.
+ *
+ * This Software is licensed under one of the following
licenses:
+ *
+ * 1) under the terms of the "Common Public License
1.0" a copy of which is
+ * in the file LICENSE.txt in the root directory.
The license is also
+ * available from the Open Source Initiative, see
+ * http://www.opensource.org/licenses/cpl.php.
+ *
+ * 2) under the terms of the "The BSD License" a copy
of which is in the file
+ * LICENSE2.txt in the root directory. The license
is also available from
+ * the Open Source Initiative, see
+ *
http://www.opensource.org/licenses/bsd-license.php.
+ *
+ * 3) under the terms of the "GNU General Public
License (GPL) Version 2" a
+ * copy of which is in the file LICENSE3.txt in the
root directory. The
+ * license is also available from the Open Source
Initiative, see
+ *
http://www.opensource.org/licenses/gpl-license.php.
+ *
+ * Licensee has the right to choose one of the above
licenses.
+ *
+ * Redistributions of source code must retain the above
copyright
+ * notice and one of the license notices.
+ *
+ * Redistributions in binary form must reproduce both
the above copyright
+ * notice, one of the license notices in the
documentation
+ * and/or other materials provided with the
distribution.
+ */
+/**********************************************************************
+ *
+ * HEADER: dat_extensions.h
+ *
+ * PURPOSE: defines the extensions to the DAT API for
uDAPL.
+ *
+ * Description: Header file for "uDAPL: User Direct
Access Programming
+ * Library, Version: 1.2"
+ *
+ * Mapping rules:
+ * All global symbols are prepended with "DAT_" or
"dat_"
+ * All DAT objects have an 'api' tag which, such
as 'ep' or 'lmr'
+ * The method table is in the provider definition
structure.
+ *
+ *
+
**********************************************************************/
+
+#ifndef _DAT_EXTENSIONS_H_
+
+extern int dat_extensions;
+
+/*
+ * Provider specific attribute strings for extension
support
+ * returned with dat_ia_query() and
+ * DAT_PROVIDER_ATTR_MASK ==
DAT_PROVIDER_FIELD_PROVIDER_SPECIFIC_ATTR
+ *
+ * DAT_NAMED_ATTR name == extended operation,
+ * value == TRUE if extended operation is
supported
+ */
+#define DAT_EXT_ATTR
"DAT_EXTENSION_INTERFACE"
+#define DAT_EXT_ATTR_RDMA_WRITE_IMMED
"DAT_EXT_RDMA_WRITE_IMMED"
+#define DAT_EXT_ATTR_RECV_IMMED
"DAT_EXT_RECV_IMMED"
+#define DAT_EXT_ATTR_RECV_IMMED_EVENT
"DAT_EXT_RECV_IMMED_EVENT"
+#define DAT_EXT_ATTR_RECV_IMMED_PAYLOAD
"DAT_EXT_RECV_IMMED_PAYLOAD"
+#define DAT_EXT_ATTR_FETCH_AND_ADD
"DAT_EXT_FETCH_AND_ADD"
+#define DAT_EXT_ATTR_CMP_AND_SWAP
"DAT_EXT_CMP_AND_SWAP"
+#define DAT_EXT_ATTR_TRUE "TRUE"
+#define DAT_EXT_ATTR_FALSE "FALSE"
+
+/*
+ * Extension OPERATIONS
+ */
+typedef enum dat_ext_op
+{
+ DAT_EXT_RDMA_WRITE_IMMED,
+ DAT_EXT_RECV_IMMED,
+ DAT_EXT_FETCH_AND_ADD,
+ DAT_EXT_CMP_AND_SWAP,
+
+} DAT_EXT_OP;
+
+/*
+ * Extension completion event TYPES
+ */
+typedef enum dat_ext_event_type
+{
+ DAT_EXT_RDMA_WRITE_IMMED_STATUS = 1,
+ DAT_EXT_RECV_NO_IMMED,
+ DAT_EXT_RECV_IMMED_DATA_EVENT,
+ DAT_EXT_RECV_IMMED_DATA_PAYLOAD,
+ DAT_EXT_FETCH_AND_ADD_STATUS,
+ DAT_EXT_CMP_AND_SWAP_STATUS,
+
+} DAT_EXT_EVENT_TYPE;
+
+/*
+ * Extension completion event DATA
+ */
+typedef struct dat_immediate_data
+{
+ DAT_UINT32 data;
+
+} DAT_RDMA_WRITE_IMMED_DATA;
+
+typedef struct dat_extension_data
+{
+ DAT_DTO_COMPLETION_EVENT_DATA dto;
+ DAT_EXT_EVENT_TYPE type;
+ union {
+ DAT_RDMA_WRITE_IMMED_DATA immed;
+ } val;
+} DAT_EXTENSION_DATA;
+
+typedef enum dat_ext_flags
+{
+ DAT_EXT_WRITE_IMMED_FLAG = 0x1,
+ DAT_EXT_WRITE_CONFIRM_FLAG = 0x2,
+
+} DAT_EXT_FLAGS;
+
+/*
+ * Extended API with redirection via DAT extension
function
+ */
+
+/*
+ * RDMA Write with IMMEDIATE extension:
+ *
+ * Asynchronous call performs a normal RDMA write to
the remote endpoint
+ * followed by a post of an extended immediate data
value to the receive
+ * EVD on the remote endpoint. Event completion for the
request completes
+ * as an DAT_EXTENSION_EVENT with type set to
DAT_EXT_RDMA_WRITE_IMMED_STATUS.
+ * Event completion on the remote endpoint completes as
an DAT_EXTENSION_EVENT
+ * with type set to DAT_EXT_RECV_IMMED_DATA_IN_EVENT or
+ * DAT_EXT_RECV_IMMED_DATA_IN_PAYLOAD depending on the
provider transport.
+ *
+ * DAT_EXT_WRITE_IMMED_FLAG requests that the supplied
+ *'immediate' value be sent as the payload of a four
byte send following
+ * the RDMA Write, or any transport-dependent
equivalent thereof.
+ * For example, on InfiniBand the request should be
translated as an
+ * RDMA Write with Immediate.
+ *
+ * DAT_EXT_WRITE_CONFIRM_FLAG requests that this DTO
+ * not complete until receipt by the far end is
confirmed.
+ *
+ * Note to Consumers: the immediate data will consume a
receive
+ * buffer at the Data Sink.
+ *
+ * Other extension flags:
+ * n/a
+ */
+#define dat_ep_post_rdma_write_immed(ep, size, lbuf,
cookie, rbuf, idata, eflgs, flgs) \
+ dat_extension( ep, \
+ DAT_EXT_RDMA_WRITE_IMMED, \
+ (size), \
+ (lbuf), \
+ (cookie), \
+ (rbuf), \
+ (idata), \
+ (eflgs), \
+ (flgs))
+
+/*
+ * Call performs a normal post receive message to the
local endpoint
+ * that includes additional 32-bit buffer space for
immediate data
+ * Event completion for the request completes as an
+ * DAT_EXTENSION_EVENT with type set to
DAT_EXT_RDMA_WRITE_IMMED_STATUS.
+ */
+#define dat_ep_post_recv_immed(ep, size, lbuf, cookie,
flgs) \
+ dat_extension( ep, \
+ DAT_EXT_RECV_IMMED, \
+ (size), \
+ (lbuf), \
+ (cookie), \
+ (flgs))
+
+/*
+ * This asynchronous call is modeled after the
InfiniBand atomic
+ * Fetch and Add operation. The add_value is added to
the 64 bit
+ * value stored at the remote memory location specified
in remote_iov
+ * and the result is stored in the local_iov.
+ */
+#define dat_ep_post_fetch_and_add(ep, add_val, lbuf,
cookie, rbuf, flgs) \
+ dat_extension( ep, \
+ DAT_EXT_FETCH_AND_ADD, \
+ (add_val), \
+ (lbuf), \
+ (cookie), \
+ (rbuf), \
+ (flgs))
+
+/*
+ * This asynchronous call is modeled after the
InfiniBand atomic
+ * Compare and Swap operation. The cmp_value is
compared to the 64 bit
+ * value stored at the remote memory location specified
in remote_iov.
+ * If the two values are equal, the 64 bit swap_value
is stored in
+ * the remote memory location. In all cases, the
original 64 bit
+ * value stored in the remote memory location is copied
to the local_iov.
+ */
+#define dat_ep_post_cmp_and_swap(ep, cmp_val, swap_val,
lbuf, cookie, rbuf, flgs) \
+ dat_extension( ep, \
+ DAT_EXT_CMP_AND_SWAP, \
+ (cmp_val), \
+ (swap_val), \
+ (lbuf), \
+ (cookie), \
+ (rbuf), \
+ (flgs))
+
+#endif /* _DAT_EXTENSIONS_H_ */
+
Index: dat/common/dat_api.c
===================================================================
--- dat/common/dat_api.c (revision 4594)
+++ dat/common/dat_api.c (working copy)
@@ -1142,6 +1142,36 @@ DAT_RETURN dat_srq_set_lw(
low_watermark);
}
+DAT_RETURN dat_extension(
+ IN DAT_HANDLE handle,
+ IN DAT_EXT_OP ext_op,
+ IN ... )
+
+{
+ DAT_RETURN status;
+ va_list args;
+
+ if (handle == NULL)
+ {
+ return DAT_ERROR(DAT_INVALID_HANDLE,
DAT_INVALID_HANDLE_EP);
+ }
+
+ /* verify provider extension support */
+ if (!dat_extensions)
+ {
+ return DAT_ERROR(DAT_NOT_IMPLEMENTED, 0);
+ }
+
+ va_start(args, ext_op);
+
+ status = DAT_EXTENSION(handle,
+ ext_op,
+ args);
+ va_end(args);
+
+ return status;
+}
+
/*
* Local variables:
* c-indent-level: 4
Index: dat/udat/udat.c
===================================================================
--- dat/udat/udat.c (revision 4594)
+++ dat/udat/udat.c (working copy)
@@ -66,6 +66,10 @@ udat_check_state ( void );
*
*
*********************************************************************/
+/*
+ * Use a global to get an unresolved when run with
pre-extension library
+ */
+int dat_extensions = 0;
/*
*
@@ -230,13 +234,44 @@ dat_ia_openv (
async_event_qlen,
async_event_handle,
ia_handle);
+
+ /*
+ * See if provider supports extensions
+ */
if (dat_status == DAT_SUCCESS)
{
- return_handle = dats_set_ia_handle (*ia_handle);
- if (return_handle >= 0)
- {
- *ia_handle = (DAT_IA_HANDLE)return_handle;
- }
+ DAT_PROVIDER_ATTR p_attr;
+ int i;
+
+ return_handle = dats_set_ia_handle
(*ia_handle);
+ if (return_handle >= 0)
+ {
+ *ia_handle = (DAT_IA_HANDLE)return_handle;
+ }
+
+ if ( dat_ia_query( *ia_handle,
+ NULL,
+ 0,
+ NULL,
+
DAT_PROVIDER_FIELD_PROVIDER_SPECIFIC_ATTR,
+ &p_attr ) == DAT_SUCCESS )
+ {
+ for ( i = 0; i <
p_attr.num_provider_specific_attr; i++ )
+ {
+ if ( (strcmp(
p_attr.provider_specific_attr[i].name,
+
"DAT_EXTENSION_INTERFACE" ) == 0) &&
+ (strcmp(
p_attr.provider_specific_attr[i].value,
+ "TRUE" ) == 0)
)
+ {
+
dat_os_dbg_print(DAT_OS_DBG_TYPE_CONSUMER_API,
+ "DAT Registry:
dat_ia_open () "
+ "DAPL
Extension Interface supported!\n");
+
+ dat_extensions = 1;
+ break;
+ }
+ }
+ }
}
return dat_status;
Index: README
===================================================================
--- README (revision 4589)
+++ README (working copy)
@@ -1,5 +1,10 @@
There are now 3 uDAPL providers for openib
(openib,openib_scm,openib_cma).
+NEW FEATURES for openib_cma provider:
+API extensions for immediate data and atomic operations
have been added.
+see dat/include/dat/dat_extensions.h for new API's.
+see dapl/test/dtest/dtest_ext.c for example test cast
+
==========
1.0 BUILD:
________________________________
YAHOO! GROUPS LINKS
* Visit your group "dat-discussions
<http://groups.yahoo.com/group/dat-discussions> " on the web.
* To unsubscribe from this group, send an email to:
dat-discussions-unsubscribe at yahoogroups.com
<mailto:dat-discussions-unsubscribe at yahoogroups.com?subject=Unsubscribe>
* Your use of Yahoo! Groups is subject to the Yahoo!
Terms of Service <http://docs.yahoo.com/info/terms/> .
________________________________
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20051223/c0a6c402/attachment.html>
More information about the general
mailing list