[openib-general] RE: [RFC][PATCH] OpenIB uDAPL extension proposal - sample immed data and atomic api's
Kanevsky, Arkady
Arkady.Kanevsky at netapp.com
Fri Dec 23 11:54:39 PST 2005
Arlin,
nice proposal, thanks.
I have one high level question and a few specific technical ones.
1. Why do you want to provide this functionality via extension instead
of part of new DAT spec, say 2.0?
This will allow Consumers to use all events, operations, and Provider/IA
functionality uniformly instead
of via 2 separate layers. This will also ensure that this basic
funcionality can be provided by all DAPL Provider
the same way on DAPL and DAT layers.
DAPL 2.0 is not done yet so we have time to incorporate that.
DAPL 2.0 already introduced new functionality which is easy to beef up
for your proposal.
See DAT_DTOS for example. DAT_EVENT is also modified to handle remote
invalidation so
a small addition for Immediate data and Atoimc ops is a sensible
addition.
This should simplify proposal significantly. As you will not need to
introduce any new
EXT structures.
In general, extension route was intended for RNIC|HCA providers to
expose HW capabilities beyond
IBTA, iWARP and VIA standards. The standard RDMA functionality is best
handle via spec addition.
DAT 2.0 does it for FMR, remote and local memory invalidation as well as
others.
I had posted a complete list of changes/addition to DAT 2.0 about a
month ago.
But we had not discussed yet version change from 1.3 to 2.0 nor how much
backwards compatibility spec
will provide.
2. What is IMMED_EVENT? is it just immediate data without any payload
one?
I suggest chnaging the name so it will not use "EVENT". Just call it
NO_PAYLOAD.
Do you want to support 2 different way to delivery immediate data?
One in event and one in data payload?
Why? I would think that just an event way will do.
3. I suggest beefing up DAT_DTO_COMPLETION_EVENT_DATA and DAT_DTOS
to convey which operation completed and return Immediate data if
complete operation had immediate data.
Since we already modified these 2 struct as part of DAT 2.0 change lets
add your proposal to the change.
This will allow Consumers to use single approach to deal with
completions, extension to the current one
but not a structural one. No need for DAT_EXTENSION_DATA,
DAT_EXT_EVENT_TYPE, DAT_EXT_OP
nor the whole mechanism for extended ops.
4. What is the purpose of DAT_EXT_WRITE_CONFIRM_FLAG? Is it to expose IB
round trip semantic?
iWARP does not support immediate data. One can try to format payload to
pass immediate data.
Is that what you had in mind?
What is the semantic meaning of the completion with this flag set?
without flag set?
Are extended flags are additonal values for COMPLETION_FLAGS? 2.4.1
talks about extended flags
but where they are passed in is not defined.
DAT 2.0 extended them already for FMR barrier. I would prefer to follow
that route rather than creating a separate
extension completion flags.
5. Why do you need RECV_IMMED? If Immed data is delivered in event no
new Recv operation is needed.
If Consumer asks for immediate data in payload where in payload will it
be?
If this is needed for local match for remote RDMA_Write to handle
immediate data lets state so.
What happens for mismatch between local and remote op? That is recv was
posted for Send and RDMA_Write
"arrived"? Vice Versa?
6. I see extension for immediate data for rdma_write but not for send.
Is this deliberate? If we are going
to extend DAT semantic to support Immediate data we can as well support
the full IBTA/iWARP functionality for it.
7. Currently memory registration do not support access to LMR or RMR by
Atomic ops.
Do you propose to extend the meaning of current MEM_PRIV for LMR and RMR
to covers atomic accesses
or add new values to LMR_MEM_PRIV and RMR_MEM_PRIV for atomic operation
support?
8. Any alignment requirements for memory used for atomic ops?
9. Any correlation requirements for SRQ buffers to support recv with
immediate data?
Have a great holidays,
Arkady
Arkady Kanevsky email: arkady at netapp.com
Network Appliance Inc. phone: 781-768-5395
1601 Trapelo Rd. - Suite 16. Fax: 781-895-1195
Waltham, MA 02451 central phone: 781-768-5300
________________________________
From: Arlin Davis [mailto:arlin.r.davis at intel.com]
Sent: Thursday, December 22, 2005 6:20 PM
To: Lentini, James; Kanevsky, Arkady
Cc: openib-general at openib.org; dat-discussions at yahoogroups.com
Subject: [RFC][PATCH] OpenIB uDAPL extension proposal - sample
immed data and atomic api's
James and Arkady,
DAPL provides a generalized abstraction to RDMA capable
transports. As a generalized abstraction, it cannot exploit the unique
properties that many of the underlying platforms/interconnects can
provide so I would like to propose a simple (minimum impact on libdat)
extensible interface to uDAPL that will allow vendors to expose such
capabilities. I am looking for feedback, especially from the DAT
collaborative. I have included both a design document and actual working
code as a reference.
The patch provides a fully tested DAT and DAPL library
(openib_cma) set with the following provider extensions:
DAT_RETURN
dat_ep_post_write_immed(
IN DAT_EP_HANDLE ep_handle,
IN DAT_COUNT num_segments
IN DAT_LMR_TRIPLET *local_iov,
IN DAT_DTO_COOKIE user_cookie,
IN DAT_RMR_TRIPLE *remote_iov,
IN DAT_UINT32 immediate_data,
IN DAT_COMPLETION_FLAGS completion_flags);
DAT_RETURN
dat_ep_post_cmp_and_swap(
IN DAT_EP_HANDLE ep_handle,
IN DAT_UINT64 cmp_value,
IN DAT_UINT64 swap_value,
IN DAT_LMR_TRIPLE *local_iov,
IN DAT_DTO_COOKIE user_cookie,
IN DAT_RMR_TRIPLE *remote_iov,
IN DAT_COMPLETION_FLAGS completion_flags);
DAT_RETURN
dat_ep_post_fetch_and_add(
IN DAT_EP_HANDLE ep_handle,
IN DAT_UINT64 add_value,
IN DAT_LMR_TRIPLE *local_iov,
IN DAT_DTO_COOKIE user_cookie,
IN DAT_RMR_TRIPLE *remote_iov,
IN DAT_COMPLETION_FLAGS completion_flags);
Also, included is a sample program (dtest_ext.c) that can be
used as a programming example.
Thanks,
-arlin
Signed-off by: Arlin Davis <ardavis at ichips.intel.com>
<mailto:ardavis at ichips.intel.com>
Index: test/dtest/dat.conf
===================================================================
--- test/dtest/dat.conf (revision 4589)
+++ test/dtest/dat.conf (working copy)
@@ -1,11 +1,20 @@
#
-# DAT 1.1 and 1.2 configuration file
+# DAT 1.2 configuration file
#
# Each entry should have the following fields:
#
# <ia_name> <api_version> <threadsafety> <default> <lib_path> \
# <provider_version> <ia_params> <platform_params>
#
-# Example for openib using the first Mellanox adapter, port 1
and port 2
+# Example for openib_cma and openib_scm
+#
+# For scm version you specify <ia_params> as actual device name
and port
+# For cma version you specify <ia_params> as:
+# network address, network hostname, or netdev name and 0
for port
+#
+OpenIB-scm1 u1.2 nonthreadsafe default
/usr/local/openib_dapl/udapl/Target/libdapl.so mv_dapl.1.2 "mthca0 1" ""
+OpenIB-scm2 u1.2 nonthreadsafe default
/usr/local/openib_dapl/udapl/Target/libdapl.so mv_dapl.1.2 "mthca0 2" ""
+OpenIB-cma-ip u1.2 nonthreadsafe default
/usr/local/openib_dapl/udapl/Target/libdapl.so mv_dapl.1.2 "192.168.0.22
0" ""
+OpenIB-cma-name u1.2 nonthreadsafe default
/usr/local/openib_dapl/udapl/Target/libdapl.so mv_dapl.1.2 "svr1-ib0 0"
""
+OpenIB-cma-netdev u1.2 nonthreadsafe default
/usr/local/openib_dapl/udapl/Target/libdapl.so mv_dapl.1.2 "ib0 0" ""
-IB1 u1.2 nonthreadsafe default
Index: test/dtest/makefile
===================================================================
--- test/dtest/makefile (revision 4589)
+++ test/dtest/makefile (working copy)
@@ -4,13 +4,18 @@ CFLAGS = -O2 -g
DAT_INC = ../../dat/include
DAT_LIB = /usr/local/lib
-all: dtest
+all: dtest dtest_ext
clean:
- rm -f *.o;touch *.c;rm -f dtest
+ rm -f *.o;touch *.c;rm -f dtest dtest_ext
dtest: ./dtest.c
$(CC) $(CFLAGS) ./dtest.c -o dtest \
-DDAPL_PROVIDER='"OpenIB-cma-ip"' \
-I $(DAT_INC) -L $(DAT_LIB) -ldat
+dtest_ext: ./dtest_ext.c
+ $(CC) $(CFLAGS) ./dtest_ext.c -o dtest_ext \
+ -DDAPL_PROVIDER='"OpenIB-cma-ip"' \
+ -I $(DAT_INC) -L $(DAT_LIB) -ldat
+
Index: test/dtest/README
===================================================================
--- test/dtest/README (revision 4589)
+++ test/dtest/README (working copy)
@@ -1,10 +1,11 @@
simple dapl test just for initial openIB uDAPL testing...
dtest/dtest.c
+ dtest/dtest_ext.c
dtest/makefile
dtest/dat.conf
-to build (default uDAPL name == IB1, ib device == mthca0,
port == 1)
+to build (default uDAPL name == OpenIB-cma-ip)
edit makefile and change path (DAT_LIB) to appropriate
libdat.so
edit dat.conf and change path to appropriate libdapl.so
cp dat.conf to /etc/dat.conf
Index: dapl/include/dapl.h
===================================================================
--- dapl/include/dapl.h (revision 4589)
+++ dapl/include/dapl.h (working copy)
@@ -1,25 +1,28 @@
/*
- * Copyright (c) 2002-2003, Network Appliance, Inc. All rights
reserved.
+ * Copyright (c) 2002-2005, Network Appliance, Inc. All rights
reserved.
*
* This Software is licensed under one of the following
licenses:
- *
+ *
* 1) under the terms of the "Common Public License 1.0" a copy
of which is
- * available from the Open Source Initiative, see
+ * in the file LICENSE.txt in the root directory. The
license is also
+ * available from the Open Source Initiative, see
* http://www.opensource.org/licenses/cpl.php.
- *
- * 2) under the terms of the "The BSD License" a copy of which
is
- * available from the Open Source Initiative, see
+ *
+ * 2) under the terms of the "The BSD License" a copy of which
is in the file
+ * LICENSE2.txt in the root directory. The license is also
available from
+ * the Open Source Initiative, see
* http://www.opensource.org/licenses/bsd-license.php.
- *
+ *
* 3) under the terms of the "GNU General Public License (GPL)
Version 2" a
- * copy of which is available from the Open Source
Initiative, see
+ * copy of which is in the file LICENSE3.txt in the root
directory. The
+ * license is also available from the Open Source
Initiative, see
* http://www.opensource.org/licenses/gpl-license.php.
- *
+ *
* Licensee has the right to choose one of the above licenses.
- *
+ *
* Redistributions of source code must retain the above
copyright
* notice and one of the license notices.
- *
+ *
* Redistributions in binary form must reproduce both the above
copyright
* notice, one of the license notices in the documentation
* and/or other materials provided with the distribution.
@@ -61,6 +64,8 @@
#include "dapl_dummy_util.h"
#elif OPENIB
#include "dapl_ib_util.h"
+#elif DET
+#include "dapl_det_util.h"
#endif
/*********************************************************************
@@ -213,6 +218,10 @@ typedef struct dapl_cookie
DAPL_COOKIE;
typedef struct dapl_dto_cookie DAPL_DTO_COOKIE;
typedef struct dapl_rmr_cookie DAPL_RMR_COOKIE;
+#ifdef DAPL_EXTENSIONS
+typedef struct dapl_ext_cookie DAPL_EXT_COOKIE;
+#endif
+
typedef struct dapl_private DAPL_PRIVATE;
typedef void (*DAPL_CONNECTION_STATE_HANDLER) (
@@ -563,6 +572,13 @@ typedef enum dapl_dto_type
DAPL_DTO_TYPE_RECV,
DAPL_DTO_TYPE_RDMA_WRITE,
DAPL_DTO_TYPE_RDMA_READ,
+#ifdef DAPL_EXTENSIONS
+ DAPL_DTO_TYPE_RDMA_WRITE_IMMED,
+ DAPL_DTO_TYPE_RECV_IMMED,
+ DAPL_DTO_TYPE_CMP_AND_SWAP,
+ DAPL_DTO_TYPE_FETCH_AND_ADD,
+#endif
+
} DAPL_DTO_TYPE;
typedef enum dapl_cookie_type
@@ -570,6 +586,9 @@ typedef enum dapl_cookie_type
DAPL_COOKIE_TYPE_NULL,
DAPL_COOKIE_TYPE_DTO,
DAPL_COOKIE_TYPE_RMR,
+#ifdef DAPL_EXTENSIONS
+ DAPL_COOKIE_TYPE_EXTENSION,
+#endif
} DAPL_COOKIE_TYPE;
/* DAPL_DTO_COOKIE used as context for DTO WQEs */
@@ -587,6 +606,27 @@ struct dapl_rmr_cookie
DAT_RMR_COOKIE cookie;
};
+#ifdef DAPL_EXTENSIONS
+
+/* DAPL extended cookie types */
+typedef enum dapl_ext_type
+{
+ DAPL_EXT_TYPE_RDMA_WRITE_IMMED,
+ DAPL_EXT_TYPE_CMP_AND_SWAP,
+ DAPL_EXT_TYPE_FETCH_AND_ADD,
+ DAPL_EXT_TYPE_RECV
+} DAPL_EXT_TYPE;
+
+/* DAPL extended cookie */
+struct dapl_ext_cookie
+{
+ DAPL_EXT_TYPE type;
+ DAT_DTO_COOKIE cookie;
+ DAT_COUNT size; /* used RDMA write with
immed */
+};
+
+#endif
+
/* DAPL_COOKIE used as context for WQEs */
struct dapl_cookie
{
@@ -597,6 +637,9 @@ struct dapl_cookie
{
DAPL_DTO_COOKIE dto;
DAPL_RMR_COOKIE rmr;
+#ifdef DAPL_EXTENSIONS
+ DAPL_EXT_COOKIE ext;
+#endif
} val;
};
@@ -1116,6 +1159,15 @@ extern DAT_RETURN dapl_srq_set_lw(
IN DAT_SRQ_HANDLE, /* srq_handle */
IN DAT_COUNT); /* low_watermark */
+#ifdef DAPL_EXTENSIONS
+
+extern DAT_RETURN dapl_extensions(
+ IN DAT_HANDLE, /* dat_handle */
+ IN DAT_EXT_OP, /* extension operation */
+ IN va_list ); /* va_list args */
+
+#endif
+
/*
* DAPL internal utility function prototpyes
*/
Index: dapl/udapl/Makefile
===================================================================
--- dapl/udapl/Makefile (revision 4589)
+++ dapl/udapl/Makefile (working copy)
@@ -156,6 +156,7 @@ PROVIDER = $(TOPDIR)/../openib_cma
CFLAGS += -DOPENIB
CFLAGS += -DCQ_WAIT_OBJECT
CFLAGS += -I/usr/local/include/infiniband
+CFLAGS += -I/usr/local/include/rdma
endif
#
@@ -168,6 +169,12 @@ endif
# VN_MEM_SHARED_VIRTUAL_SUPPORT
# CFLAGS += -DVN_MEM_SHARED_VIRTUAL_SUPPORT=1
+# If an implementation supports DAPL extensions
+CFLAGS += -DDAPL_EXTENSIONS
+
+# If an implementation supports DAPL provider specific
attributes
+CFLAGS += -DDAPL_PROVIDER_SPECIFIC_ATTR
+
CFLAGS += -I.
CFLAGS += -I..
CFLAGS += -I../../dat/include
@@ -283,6 +290,8 @@ LDFLAGS += -libverbs -lrdmacm
LDFLAGS += -rpath /usr/local/lib -L /usr/local/lib
PROVIDER_SRCS = dapl_ib_util.c dapl_ib_cq.c dapl_ib_qp.c \
dapl_ib_cm.c dapl_ib_mem.c
+# implementation supports DAPL extensions
+PROVIDER_SRCS += dapl_ib_extensions.c
endif
UDAPL_SRCS = dapl_init.c \
Index: dapl/common/dapl_ia_query.c
===================================================================
--- dapl/common/dapl_ia_query.c (revision 4589)
+++ dapl/common/dapl_ia_query.c (working copy)
@@ -167,6 +167,14 @@ dapl_ia_query (
#if !defined(__KDAPL__)
provider_attr->pz_support = DAT_PZ_UNIQUE;
#endif /* !KDAPL */
+
+ /*
+ * Have provider set their own.
+ */
+#ifdef DAPL_PROVIDER_SPECIFIC_ATTR
+ dapls_set_provider_specific_attr(provider_attr);
+#endif
+
/*
* Set up evd_stream_merging_supported options. Note there
is
* one bit per allowable combination, using the ordinal
Index: dapl/common/dapl_adapter_util.h
===================================================================
--- dapl/common/dapl_adapter_util.h (revision 4589)
+++ dapl/common/dapl_adapter_util.h (working copy)
@@ -256,6 +256,21 @@ dapls_ib_wait_object_wait (
IN u_int32_t timeout);
#endif
+#ifdef DAPL_PROVIDER_SPECIFIC_ATTR
+void
+dapls_set_provider_specific_attr(
+ IN DAT_PROVIDER_ATTR *provider_attr );
+#endif
+
+#ifdef DAPL_EXTENSIONS
+void
+dapls_cqe_to_event_extension(
+ IN DAPL_EP *ep_ptr,
+ IN DAPL_COOKIE *cookie,
+ IN ib_work_completion_t *cqe_ptr,
+ OUT DAT_EVENT *event_ptr);
+#endif
+
/*
* Values for provider DAT_NAMED_ATTR
*/
@@ -272,6 +287,8 @@ dapls_ib_wait_object_wait (
#include "dapl_dummy_dto.h"
#elif OPENIB
#include "dapl_ib_dto.h"
+#elif DET
+#include "dapl_det_dto.h"
#endif
Index: dapl/common/dapl_provider.c
===================================================================
--- dapl/common/dapl_provider.c (revision 4589)
+++ dapl/common/dapl_provider.c (working copy)
@@ -221,7 +221,11 @@ DAT_PROVIDER g_dapl_provider_template =
&dapl_srq_post_recv,
&dapl_srq_query,
&dapl_srq_resize,
- &dapl_srq_set_lw
+ &dapl_srq_set_lw,
+
+#ifdef DAPL_EXTENSIONS
+ &dapl_extensions
+#endif
};
#endif /* __KDAPL__ */
Index: dapl/common/dapl_evd_util.c
===================================================================
--- dapl/common/dapl_evd_util.c (revision 4589)
+++ dapl/common/dapl_evd_util.c (working copy)
@@ -502,6 +502,20 @@ dapli_evd_eh_print_cqe (
#ifdef DAPL_DBG
static char *optable[] =
{
+#ifdef OPENIB
+ /* different order for openib verbs */
+ "OP_RDMA_WRITE",
+ "OP_RDMA_WRITE_IMM",
+ "OP_SEND",
+ "OP_SEND_IMM",
+ "OP_RDMA_READ",
+ "OP_COMP_AND_SWAP",
+ "OP_FETCH_AND_ADD",
+ "OP_RECEIVE",
+ "OP_RECEIVE_IMM",
+ "OP_BIND_MW",
+ "OP_INVALID",
+#else
"OP_SEND",
"OP_RDMA_READ",
"OP_RDMA_WRITE",
@@ -509,6 +523,7 @@ dapli_evd_eh_print_cqe (
"OP_FETCH_AND_ADD",
"OP_RECEIVE",
"OP_BIND_MW",
+#endif
0
};
@@ -1113,6 +1128,15 @@ dapli_evd_cqe_to_event (
dapls_cookie_dealloc (&ep_ptr->req_buffer, cookie);
break;
}
+
+#ifdef DAPL_EXTENSIONS
+ case DAPL_COOKIE_TYPE_EXTENSION:
+ {
+ dapls_cqe_to_event_extension(ep_ptr, cookie, cqe_ptr,
event_ptr);
+ break;
+ }
+#endif
+
default:
{
dapl_os_assert (!"Invalid Operation type");
Index: dapl/openib_cma/dapl_ib_dto.h
===================================================================
--- dapl/openib_cma/dapl_ib_dto.h (revision 4589)
+++ dapl/openib_cma/dapl_ib_dto.h (working copy)
@@ -35,7 +35,7 @@
*
* Description:
*
- * The uDAPL openib provider - DTO operations and CQE macros
+ * The OpenIB uCMA provider - DTO operations and CQE macros
*
************************************************************************
****
* Source Control System Information
@@ -119,7 +119,6 @@ dapls_ib_post_recv (
return DAT_SUCCESS;
}
-
/*
* dapls_ib_post_send
*
@@ -191,7 +190,7 @@ dapls_ib_post_send (
if (cookie != NULL)
cookie->val.dto.size = total_len;
-
+
if ((op_type == OP_RDMA_WRITE) || (op_type ==
OP_RDMA_READ)) {
wr.wr.rdma.remote_addr = remote_iov->target_address;
wr.wr.rdma.rkey = remote_iov->rmr_context;
@@ -224,6 +223,152 @@ dapls_ib_post_send (
return DAT_SUCCESS;
}
+#ifdef DAPL_EXTENSIONS
+/*
+ * dapls_ib_post_ext_send
+ *
+ * Provider specific extended Post SEND function
+ */
+STATIC _INLINE_ DAT_RETURN
+dapls_ib_post_ext_send (
+ IN DAPL_EP *ep_ptr,
+ IN ib_send_op_type_t op_type,
+ IN DAPL_COOKIE *cookie,
+ IN DAT_COUNT segments,
+ IN DAT_LMR_TRIPLET *local_iov,
+ IN const DAT_RMR_TRIPLET *remote_iov,
+ IN DAT_UINT32 idata,
+ IN DAT_UINT64 compare_add,
+ IN DAT_UINT64 swap,
+ IN DAT_COMPLETION_FLAGS completion_flags)
+{
+ dapl_dbg_log(DAPL_DBG_TYPE_EP,
+ " post_snd: ep %p op %d ck %p sgs",
+ "%d l_iov %p r_iov %p f %d\n",
+ ep_ptr, op_type, cookie, segments, local_iov,
+ remote_iov, completion_flags);
+
+ ib_data_segment_t ds_array[DEFAULT_DS_ENTRIES];
+ ib_data_segment_t *ds_array_p;
+ struct ibv_send_wr wr;
+ struct ibv_send_wr *bad_wr;
+ ib_hca_transport_t *ibt_ptr =
+ &ep_ptr->header.owner_ia->hca_ptr->ib_trans;
+ DAT_COUNT i, total_len;
+
+ dapl_dbg_log(DAPL_DBG_TYPE_EP,
+ " post_snd: ep %p cookie %p segs %d l_iov
%p\n",
+ ep_ptr, cookie, segments, local_iov);
+
+ if(segments <= DEFAULT_DS_ENTRIES)
+ ds_array_p = ds_array;
+ else
+ ds_array_p =
+ dapl_os_alloc(segments *
sizeof(ib_data_segment_t));
+
+ if (NULL == ds_array_p)
+ return (DAT_INSUFFICIENT_RESOURCES);
+
+ /* setup the work request */
+ wr.next = 0;
+ wr.opcode = op_type;
+ wr.num_sge = 0;
+ wr.send_flags = 0;
+ wr.wr_id = (uint64_t)(uintptr_t)cookie;
+ wr.sg_list = ds_array_p;
+ total_len = 0;
+
+ for (i = 0; i < segments; i++ ) {
+ if ( !local_iov[i].segment_length )
+ continue;
+
+ ds_array_p->addr = (uint64_t)
local_iov[i].virtual_address;
+ ds_array_p->length = local_iov[i].segment_length;
+ ds_array_p->lkey = local_iov[i].lmr_context;
+
+ dapl_dbg_log(DAPL_DBG_TYPE_EP,
+ " post_snd: lkey 0x%x va %p len %d\n",
+ ds_array_p->lkey, ds_array_p->addr,
+ ds_array_p->length );
+
+ total_len += ds_array_p->length;
+ wr.num_sge++;
+ ds_array_p++;
+ }
+
+ if (cookie != NULL)
+ cookie->val.dto.size = total_len;
+
+ if ((op_type == OP_RDMA_WRITE) ||
+ (op_type == OP_RDMA_WRITE_IMM) ||
+ (op_type == OP_RDMA_READ)) {
+ wr.wr.rdma.remote_addr = remote_iov->target_address;
+ wr.wr.rdma.rkey = remote_iov->rmr_context;
+ dapl_dbg_log(DAPL_DBG_TYPE_EP,
+ " post_snd_rdma: rkey 0x%x va %#016Lx\n",
+ wr.wr.rdma.rkey, wr.wr.rdma.remote_addr);
+ }
+
+ switch (op_type) {
+ case OP_RDMA_WRITE_IMM:
+ dapl_dbg_log(DAPL_DBG_TYPE_EP,
+ " post_snd: OP_RDMA_WRITE_IMMED=0x%x\n",
idata );
+ wr.imm_data = idata;
+ break;
+ case OP_COMP_AND_SWAP:
+ /* OP_COMP_AND_SWAP has direct IBAL wr_type mapping
*/
+ dapl_dbg_log(DAPL_DBG_TYPE_EP,
+ " post_snd: OP_COMP_AND_SWAP=%lx,"
+ "%lx rkey 0x%x va %#016Lx\n",
+ compare_add, swap,
remote_iov->rmr_context,
+ remote_iov->target_address);
+
+ wr.wr.atomic.compare_add = compare_add;
+ wr.wr.atomic.swap = swap;
+ wr.wr.atomic.remote_addr =
remote_iov->target_address;
+ wr.wr.atomic.rkey = remote_iov->rmr_context;
+ break;
+ case OP_FETCH_AND_ADD:
+ /* OP_FETCH_AND_ADD has direct IBAL wr_type mapping
*/
+ dapl_dbg_log(DAPL_DBG_TYPE_EP,
+ " post_snd: OP_FETCH_AND_ADD=%lx,"
+ "%lx rkey 0x%x va %#016Lx\n",
+ compare_add, remote_iov->rmr_context,
+ remote_iov->target_address);
+
+ wr.wr.atomic.compare_add = compare_add;
+ wr.wr.atomic.remote_addr =
remote_iov->target_address;
+ wr.wr.atomic.rkey = remote_iov->rmr_context;
+ break;
+ default:
+ break;
+ }
+
+ /* inline data for send or write ops */
+ if ((total_len <= ibt_ptr->max_inline_send) &&
+ ((op_type == OP_SEND) || (op_type == OP_RDMA_WRITE)))
+ wr.send_flags |= IBV_SEND_INLINE;
+
+ /* set completion flags in work request */
+ wr.send_flags |= (DAT_COMPLETION_SUPPRESS_FLAG &
+ completion_flags) ? 0 :
IBV_SEND_SIGNALED;
+ wr.send_flags |= (DAT_COMPLETION_BARRIER_FENCE_FLAG &
+ completion_flags) ? IBV_SEND_FENCE : 0;
+ wr.send_flags |= (DAT_COMPLETION_SOLICITED_WAIT_FLAG &
+ completion_flags) ? IBV_SEND_SOLICITED :
0;
+
+ dapl_dbg_log(DAPL_DBG_TYPE_EP,
+ " post_snd: op 0x%x flags 0x%x sglist %p,
%d\n",
+ wr.opcode, wr.send_flags, wr.sg_list,
wr.num_sge);
+
+ if (ibv_post_send(ep_ptr->qp_handle->cm_id->qp, &wr,
&bad_wr))
+ return( dapl_convert_errno(EFAULT,"ibv_recv") );
+
+ dapl_dbg_log(DAPL_DBG_TYPE_EP," post_snd: returned\n");
+ return DAT_SUCCESS;
+}
+#endif
+
STATIC _INLINE_ DAT_RETURN
dapls_ib_optional_prv_dat(
IN DAPL_CR *cr_ptr,
Index: dapl/openib_cma/dapl_ib_util.c
===================================================================
--- dapl/openib_cma/dapl_ib_util.c (revision 4589)
+++ dapl/openib_cma/dapl_ib_util.c (working copy)
@@ -35,7 +35,7 @@
*
* Description:
*
- * The uDAPL openib provider - init, open, close, utilities,
work thread
+ * The OpenIB uCMA provider - init, open, close, utilities,
work thread
*
************************************************************************
****
* Source Control System Information
@@ -64,7 +64,6 @@ static const char rcsid[] = "$Id: $";
#include <net/if.h> /* for struct ifreq */
#include <net/if_arp.h> /* for ARPHRD_INFINIBAND */
-
int g_dapl_loopback_connection = 0;
int g_ib_pipe[2];
ib_thread_state_t g_ib_thread_state = 0;
@@ -727,7 +726,7 @@ void dapli_thread(void *arg)
int ret,idx,fds;
char rbuf[2];
- dapl_dbg_log (DAPL_DBG_TYPE_CM,
+ dapl_dbg_log (DAPL_DBG_TYPE_UTIL,
" ib_thread(%d,0x%x): ENTER: pipe %d ucma
%d\n",
getpid(), g_ib_thread, g_ib_pipe[0],
rdma_get_fd());
@@ -767,7 +766,7 @@ void dapli_thread(void *arg)
ufds[idx].revents = 0;
uhca[idx] = hca;
- dapl_dbg_log(DAPL_DBG_TYPE_CM,
+ dapl_dbg_log(DAPL_DBG_TYPE_UTIL,
" ib_thread(%d) poll_fd: hca[%d]=%p,
async=%d"
" pipe=%d cm=%d cq=d\n",
getpid(), hca, ufds[idx-1].fd,
@@ -783,14 +782,14 @@ void dapli_thread(void *arg)
dapl_os_unlock(&g_hca_lock);
ret = poll(ufds, fds, -1);
if (ret <= 0) {
- dapl_dbg_log(DAPL_DBG_TYPE_WARN,
+ dapl_dbg_log(DAPL_DBG_TYPE_UTIL,
" ib_thread(%d): ERR %s poll\n",
getpid(),strerror(errno));
dapl_os_lock(&g_hca_lock);
continue;
}
- dapl_dbg_log(DAPL_DBG_TYPE_CM,
+ dapl_dbg_log(DAPL_DBG_TYPE_UTIL,
" ib_thread(%d) poll_event: "
" async=0x%x pipe=0x%x cm=0x%x cq=0x%x\n",
getpid(), ufds[idx-1].revents,
ufds[0].revents,
@@ -834,3 +833,63 @@ void dapli_thread(void *arg)
dapl_os_unlock(&g_hca_lock);
}
+#ifdef DAPL_PROVIDER_SPECIFIC_ATTR
+/*
+ * dapls_set_provider_specific_attr
+ *
+ * Input:
+ * attr_ptr Pointer provider attributes
+ *
+ * Output:
+ * none
+ *
+ * Returns:
+ * void
+ */
+DAT_NAMED_ATTR ib_attrs[] = {
+
+#ifdef DAPL_EXTENSIONS
+ {
+ DAT_EXT_ATTR,
+ DAT_EXT_ATTR_TRUE
+ },
+ {
+ DAT_EXT_ATTR_RDMA_WRITE_IMMED,
+ DAT_EXT_ATTR_TRUE
+ },
+ {
+ DAT_EXT_ATTR_RECV_IMMED,
+ DAT_EXT_ATTR_TRUE
+ },
+ /* inbound immediate data placed in event, NOT payload */
+ {
+ DAT_EXT_ATTR_RECV_IMMED_EVENT,
+ DAT_EXT_ATTR_TRUE
+ },
+ {
+ DAT_EXT_ATTR_FETCH_AND_ADD,
+ DAT_EXT_ATTR_TRUE
+ },
+ {
+ DAT_EXT_ATTR_CMP_AND_SWAP,
+ DAT_EXT_ATTR_TRUE
+ },
+#else
+ {
+ "DAT_EXTENSION_INTERFACE",
+ "FALSE"
+ },
+#endif
+};
+
+#define SPEC_ATTR_SIZE(x) ( sizeof(x)/sizeof(DAT_NAMED_ATTR) )
+
+void dapls_set_provider_specific_attr(
+ IN DAT_PROVIDER_ATTR *attr_ptr )
+{
+ attr_ptr->num_provider_specific_attr =
SPEC_ATTR_SIZE(ib_attrs);
+ attr_ptr->provider_specific_attr = ib_attrs;
+}
+
+#endif
+
Index: dapl/openib_cma/dapl_ib_mem.c
===================================================================
--- dapl/openib_cma/dapl_ib_mem.c (revision 4589)
+++ dapl/openib_cma/dapl_ib_mem.c (working copy)
@@ -25,9 +25,9 @@
/**********************************************************************
*
- * MODULE: dapl_det_mem.c
+ * MODULE: dapl_ib_mem.c
*
- * PURPOSE: Intel DET APIs: Memory windows, registration,
+ * PURPOSE: OpenIB uCMA provider Memory windows, registration,
* and protection domain
*
* $Id: $
@@ -72,12 +72,10 @@ dapls_convert_privileges(IN DAT_MEM_PRIV
access |= IBV_ACCESS_LOCAL_WRITE;
if (DAT_MEM_PRIV_REMOTE_WRITE_FLAG & privileges)
access |= IBV_ACCESS_REMOTE_WRITE;
- if (DAT_MEM_PRIV_REMOTE_READ_FLAG & privileges)
- access |= IBV_ACCESS_REMOTE_READ;
- if (DAT_MEM_PRIV_REMOTE_READ_FLAG & privileges)
- access |= IBV_ACCESS_REMOTE_READ;
- if (DAT_MEM_PRIV_REMOTE_READ_FLAG & privileges)
+ if (DAT_MEM_PRIV_REMOTE_READ_FLAG & privileges) {
access |= IBV_ACCESS_REMOTE_READ;
+ access |= IBV_ACCESS_REMOTE_ATOMIC;
+ }
return access;
}
Index: dapl/openib_cma/dapl_ib_cm.c
===================================================================
--- dapl/openib_cma/dapl_ib_cm.c (revision 4589)
+++ dapl/openib_cma/dapl_ib_cm.c (working copy)
@@ -35,7 +35,7 @@
*
* Description:
*
- * The uDAPL openib provider - connection management
+ * The OpenIB uCMA provider - uCMA connection management
*
************************************************************************
****
* Source Control System Information
@@ -592,7 +592,11 @@ dapls_ib_setup_conn_listener(IN DAPL_IA
if (rdma_bind_addr(conn->cm_id,
(struct sockaddr
*)&ia_ptr->hca_ptr->hca_address)) {
- dat_status =
dapl_convert_errno(errno,"setup_listener");
+ if (errno == -EBUSY)
+ dat_status = DAT_CONN_QUAL_IN_USE;
+ else
+ dat_status =
+
dapl_convert_errno(errno,"setup_listener");
goto bail;
}
Index: dapl/openib_cma/dapl_ib_qp.c
===================================================================
--- dapl/openib_cma/dapl_ib_qp.c (revision 4589)
+++ dapl/openib_cma/dapl_ib_qp.c (working copy)
@@ -25,9 +25,9 @@
/**********************************************************************
*
- * MODULE: dapl_det_qp.c
+ * MODULE: dapl_ib_qp.c
*
- * PURPOSE: QP routines for access to DET Verbs
+ * PURPOSE: OpenIB uCMA QP routines
*
* $Id: $
**********************************************************************/
Index: dapl/openib_cma/dapl_ib_util.h
===================================================================
--- dapl/openib_cma/dapl_ib_util.h (revision 4589)
+++ dapl/openib_cma/dapl_ib_util.h (working copy)
@@ -35,7 +35,7 @@
*
* Description:
*
- * The uDAPL openib provider - definitions, prototypes,
+ * The OpenIB uCMA provider - definitions, prototypes,
*
************************************************************************
****
* Source Control System Information
Index: dapl/openib_cma/README
===================================================================
--- dapl/openib_cma/README (revision 4589)
+++ dapl/openib_cma/README (working copy)
@@ -23,15 +23,22 @@ New files for openib_scm provider
dapl/openib_cma/dapl_ib_util.c
dapl/openib_cma/dapl_ib_util.h
dapl/openib_cma/dapl_ib_cm.c
+ dapl/openib_cma/dapl_ib_extensions.c
A simple dapl test just for openib_scm testing...
test/dtest/dtest.c
+ test/dtest/dtest_ext.c
test/dtest/makefile
server: dtest -s
client: dtest -h hostname
+or with extensions
+
+ server: dtest_ext -s
+ client: dtest_ext -h hostname
+
known issues:
no memory windows support in ibverbs, dat_create_rmr
fails.
Index: dapl/openib_cma/dapl_ib_cq.c
===================================================================
--- dapl/openib_cma/dapl_ib_cq.c (revision 4589)
+++ dapl/openib_cma/dapl_ib_cq.c (working copy)
@@ -35,7 +35,7 @@
*
* Description:
*
- * The uDAPL openib provider - completion queue
+ * The OpenIB uCMA provider - completion queue
*
************************************************************************
****
* Source Control System Information
@@ -498,7 +498,10 @@ dapls_ib_wait_object_wait(IN ib_wait_obj
if (timeout != DAT_TIMEOUT_INFINITE)
timeout_ms = timeout/1000;
- status = poll(&cq_fd, 1, timeout_ms);
+ /* restart syscall */
+ while ((status = poll(&cq_fd, 1, timeout_ms)) == -1 )
+ if (errno == EINTR)
+ continue;
/* returned event */
if (status > 0) {
@@ -511,6 +514,8 @@ dapls_ib_wait_object_wait(IN ib_wait_obj
/* timeout */
} else if (status == 0)
status = ETIMEDOUT;
+ else
+ status = errno;
dapl_dbg_log(DAPL_DBG_TYPE_UTIL,
" cq_object_wait: RET evd %p ibv_cq %p ibv_ctx
%p %s\n",
Index: dat/include/dat/dat_redirection.h
===================================================================
--- dat/include/dat/dat_redirection.h (revision 4589)
+++ dat/include/dat/dat_redirection.h (working copy)
@@ -59,10 +59,10 @@ typedef struct dat_provider DAT_PROVIDER
* This would allow a good compiler to avoid indirection
overhead when
* making function calls.
*/
-
#define DAT_HANDLE_TO_PROVIDER(handle) (*(DAT_PROVIDER
**)(handle))
#endif
+
#define DAT_IA_QUERY(ia, evd, ia_msk, ia_ptr, p_msk, p_ptr) \
(*DAT_HANDLE_TO_PROVIDER (ia)->ia_query_func) (\
(ia), \
@@ -395,6 +395,12 @@ typedef struct dat_provider DAT_PROVIDER
(lbuf), \
(cookie))
+#define DAT_EXTENSION(handle, op, args) \
+ (*DAT_HANDLE_TO_PROVIDER (handle)->extension_func) (\
+ (handle), \
+ (op), \
+ (args))
+
/***************************************************************
*
* FUNCTION PROTOTYPES
@@ -720,4 +726,11 @@ typedef DAT_RETURN (*DAT_SRQ_POST_RECV_F
IN DAT_LMR_TRIPLET *, /* local_iov */
IN DAT_DTO_COOKIE ); /* user_cookie */
+/* Extension function */
+#include <stdarg.h>
+typedef DAT_RETURN (*DAT_EXTENSION_FUNC) (
+ IN DAT_HANDLE, /* dat handle */
+ IN DAT_EXT_OP, /* extension operation */
+ IN va_list ); /* va_list */
+
#endif /* _DAT_REDIRECTION_H_ */
Index: dat/include/dat/dat.h
===================================================================
--- dat/include/dat/dat.h (revision 4589)
+++ dat/include/dat/dat.h (working copy)
@@ -854,11 +854,15 @@ typedef enum dat_event_number
DAT_ASYNC_ERROR_EP_BROKEN = 0x08003,
DAT_ASYNC_ERROR_TIMED_OUT = 0x08004,
DAT_ASYNC_ERROR_PROVIDER_INTERNAL_ERROR = 0x08005,
- DAT_SOFTWARE_EVENT = 0x10001
+ DAT_SOFTWARE_EVENT = 0x10001,
+ DAT_EXTENSION_EVENT = 0x20001
+
} DAT_EVENT_NUMBER;
-/* Union for event Data */
+/* include extension data definitions */
+#include <dat/dat_extensions.h>
+/* Union for event Data */
typedef union dat_event_data
{
DAT_DTO_COMPLETION_EVENT_DATA
dto_completion_event_data;
@@ -867,6 +871,7 @@ typedef union dat_event_data
DAT_CONNECTION_EVENT_DATA connect_event_data;
DAT_ASYNCH_ERROR_EVENT_DATA asynch_error_event_data;
DAT_SOFTWARE_EVENT_DATA software_event_data;
+ DAT_EXTENSION_DATA extension_data;
} DAT_EVENT_DATA;
/* Event struct that holds all event information */
@@ -1222,6 +1227,11 @@ extern DAT_RETURN dat_srq_set_lw (
IN DAT_SRQ_HANDLE, /* srq_handle */
IN DAT_COUNT); /* low_watermark */
+extern DAT_RETURN dat_extension(
+ IN DAT_HANDLE,
+ IN DAT_EXT_OP,
+ IN ... );
+
/*
* DAT registry functions.
*
Index: dat/include/dat/udat_redirection.h
===================================================================
--- dat/include/dat/udat_redirection.h (revision 4589)
+++ dat/include/dat/udat_redirection.h (working copy)
@@ -199,7 +199,6 @@ typedef DAT_RETURN (*DAT_EVD_SET_UNWAITA
typedef DAT_RETURN (*DAT_EVD_CLEAR_UNWAITABLE_FUNC) (
IN DAT_EVD_HANDLE); /* evd_handle */
-
#include <dat/dat_redirection.h>
struct dat_provider
@@ -294,6 +293,10 @@ struct dat_provider
DAT_SRQ_QUERY_FUNC srq_query_func;
DAT_SRQ_RESIZE_FUNC srq_resize_func;
DAT_SRQ_SET_LW_FUNC srq_set_lw_func;
+
+ /* extension for provder specific functions */
+ DAT_EXTENSION_FUNC extension_func;
+
};
#endif /* _UDAT_REDIRECTION_H_ */
Index: dat/include/dat/dat_extensions.h
===================================================================
--- dat/include/dat/dat_extensions.h (revision 0)
+++ dat/include/dat/dat_extensions.h (revision 0)
@@ -0,0 +1,209 @@
+/*
+ * Copyright (c) 2002-2005, Network Appliance, Inc. All rights
reserved.
+ *
+ * This Software is licensed under one of the following
licenses:
+ *
+ * 1) under the terms of the "Common Public License 1.0" a copy
of which is
+ * in the file LICENSE.txt in the root directory. The
license is also
+ * available from the Open Source Initiative, see
+ * http://www.opensource.org/licenses/cpl.php.
+ *
+ * 2) under the terms of the "The BSD License" a copy of which
is in the file
+ * LICENSE2.txt in the root directory. The license is also
available from
+ * the Open Source Initiative, see
+ * http://www.opensource.org/licenses/bsd-license.php.
+ *
+ * 3) under the terms of the "GNU General Public License (GPL)
Version 2" a
+ * copy of which is in the file LICENSE3.txt in the root
directory. The
+ * license is also available from the Open Source
Initiative, see
+ * http://www.opensource.org/licenses/gpl-license.php.
+ *
+ * Licensee has the right to choose one of the above licenses.
+ *
+ * Redistributions of source code must retain the above
copyright
+ * notice and one of the license notices.
+ *
+ * Redistributions in binary form must reproduce both the above
copyright
+ * notice, one of the license notices in the documentation
+ * and/or other materials provided with the distribution.
+ */
+/**********************************************************************
+ *
+ * HEADER: dat_extensions.h
+ *
+ * PURPOSE: defines the extensions to the DAT API for uDAPL.
+ *
+ * Description: Header file for "uDAPL: User Direct Access
Programming
+ * Library, Version: 1.2"
+ *
+ * Mapping rules:
+ * All global symbols are prepended with "DAT_" or "dat_"
+ * All DAT objects have an 'api' tag which, such as 'ep'
or 'lmr'
+ * The method table is in the provider definition
structure.
+ *
+ *
+
**********************************************************************/
+
+#ifndef _DAT_EXTENSIONS_H_
+
+extern int dat_extensions;
+
+/*
+ * Provider specific attribute strings for extension support
+ * returned with dat_ia_query() and
+ * DAT_PROVIDER_ATTR_MASK ==
DAT_PROVIDER_FIELD_PROVIDER_SPECIFIC_ATTR
+ *
+ * DAT_NAMED_ATTR name == extended operation,
+ * value == TRUE if extended operation is
supported
+ */
+#define DAT_EXT_ATTR "DAT_EXTENSION_INTERFACE"
+#define DAT_EXT_ATTR_RDMA_WRITE_IMMED
"DAT_EXT_RDMA_WRITE_IMMED"
+#define DAT_EXT_ATTR_RECV_IMMED "DAT_EXT_RECV_IMMED"
+#define DAT_EXT_ATTR_RECV_IMMED_EVENT
"DAT_EXT_RECV_IMMED_EVENT"
+#define DAT_EXT_ATTR_RECV_IMMED_PAYLOAD
"DAT_EXT_RECV_IMMED_PAYLOAD"
+#define DAT_EXT_ATTR_FETCH_AND_ADD "DAT_EXT_FETCH_AND_ADD"
+#define DAT_EXT_ATTR_CMP_AND_SWAP "DAT_EXT_CMP_AND_SWAP"
+#define DAT_EXT_ATTR_TRUE "TRUE"
+#define DAT_EXT_ATTR_FALSE "FALSE"
+
+/*
+ * Extension OPERATIONS
+ */
+typedef enum dat_ext_op
+{
+ DAT_EXT_RDMA_WRITE_IMMED,
+ DAT_EXT_RECV_IMMED,
+ DAT_EXT_FETCH_AND_ADD,
+ DAT_EXT_CMP_AND_SWAP,
+
+} DAT_EXT_OP;
+
+/*
+ * Extension completion event TYPES
+ */
+typedef enum dat_ext_event_type
+{
+ DAT_EXT_RDMA_WRITE_IMMED_STATUS = 1,
+ DAT_EXT_RECV_NO_IMMED,
+ DAT_EXT_RECV_IMMED_DATA_EVENT,
+ DAT_EXT_RECV_IMMED_DATA_PAYLOAD,
+ DAT_EXT_FETCH_AND_ADD_STATUS,
+ DAT_EXT_CMP_AND_SWAP_STATUS,
+
+} DAT_EXT_EVENT_TYPE;
+
+/*
+ * Extension completion event DATA
+ */
+typedef struct dat_immediate_data
+{
+ DAT_UINT32 data;
+
+} DAT_RDMA_WRITE_IMMED_DATA;
+
+typedef struct dat_extension_data
+{
+ DAT_DTO_COMPLETION_EVENT_DATA dto;
+ DAT_EXT_EVENT_TYPE type;
+ union {
+ DAT_RDMA_WRITE_IMMED_DATA immed;
+ } val;
+} DAT_EXTENSION_DATA;
+
+typedef enum dat_ext_flags
+{
+ DAT_EXT_WRITE_IMMED_FLAG = 0x1,
+ DAT_EXT_WRITE_CONFIRM_FLAG = 0x2,
+
+} DAT_EXT_FLAGS;
+
+/*
+ * Extended API with redirection via DAT extension function
+ */
+
+/*
+ * RDMA Write with IMMEDIATE extension:
+ *
+ * Asynchronous call performs a normal RDMA write to the remote
endpoint
+ * followed by a post of an extended immediate data value to
the receive
+ * EVD on the remote endpoint. Event completion for the request
completes
+ * as an DAT_EXTENSION_EVENT with type set to
DAT_EXT_RDMA_WRITE_IMMED_STATUS.
+ * Event completion on the remote endpoint completes as an
DAT_EXTENSION_EVENT
+ * with type set to DAT_EXT_RECV_IMMED_DATA_IN_EVENT or
+ * DAT_EXT_RECV_IMMED_DATA_IN_PAYLOAD depending on the provider
transport.
+ *
+ * DAT_EXT_WRITE_IMMED_FLAG requests that the supplied
+ *'immediate' value be sent as the payload of a four byte send
following
+ * the RDMA Write, or any transport-dependent equivalent
thereof.
+ * For example, on InfiniBand the request should be translated
as an
+ * RDMA Write with Immediate.
+ *
+ * DAT_EXT_WRITE_CONFIRM_FLAG requests that this DTO
+ * not complete until receipt by the far end is confirmed.
+ *
+ * Note to Consumers: the immediate data will consume a receive
+ * buffer at the Data Sink.
+ *
+ * Other extension flags:
+ * n/a
+ */
+#define dat_ep_post_rdma_write_immed(ep, size, lbuf, cookie,
rbuf, idata, eflgs, flgs) \
+ dat_extension( ep, \
+ DAT_EXT_RDMA_WRITE_IMMED, \
+ (size), \
+ (lbuf), \
+ (cookie), \
+ (rbuf), \
+ (idata), \
+ (eflgs), \
+ (flgs))
+
+/*
+ * Call performs a normal post receive message to the local
endpoint
+ * that includes additional 32-bit buffer space for immediate
data
+ * Event completion for the request completes as an
+ * DAT_EXTENSION_EVENT with type set to
DAT_EXT_RDMA_WRITE_IMMED_STATUS.
+ */
+#define dat_ep_post_recv_immed(ep, size, lbuf, cookie, flgs) \
+ dat_extension( ep, \
+ DAT_EXT_RECV_IMMED, \
+ (size), \
+ (lbuf), \
+ (cookie), \
+ (flgs))
+
+/*
+ * This asynchronous call is modeled after the InfiniBand
atomic
+ * Fetch and Add operation. The add_value is added to the 64
bit
+ * value stored at the remote memory location specified in
remote_iov
+ * and the result is stored in the local_iov.
+ */
+#define dat_ep_post_fetch_and_add(ep, add_val, lbuf, cookie,
rbuf, flgs) \
+ dat_extension( ep, \
+ DAT_EXT_FETCH_AND_ADD, \
+ (add_val), \
+ (lbuf), \
+ (cookie), \
+ (rbuf), \
+ (flgs))
+
+/*
+ * This asynchronous call is modeled after the InfiniBand
atomic
+ * Compare and Swap operation. The cmp_value is compared to the
64 bit
+ * value stored at the remote memory location specified in
remote_iov.
+ * If the two values are equal, the 64 bit swap_value is stored
in
+ * the remote memory location. In all cases, the original 64
bit
+ * value stored in the remote memory location is copied to the
local_iov.
+ */
+#define dat_ep_post_cmp_and_swap(ep, cmp_val, swap_val, lbuf,
cookie, rbuf, flgs) \
+ dat_extension( ep, \
+ DAT_EXT_CMP_AND_SWAP, \
+ (cmp_val), \
+ (swap_val), \
+ (lbuf), \
+ (cookie), \
+ (rbuf), \
+ (flgs))
+
+#endif /* _DAT_EXTENSIONS_H_ */
+
Index: dat/common/dat_api.c
===================================================================
--- dat/common/dat_api.c (revision 4594)
+++ dat/common/dat_api.c (working copy)
@@ -1142,6 +1142,36 @@ DAT_RETURN dat_srq_set_lw(
low_watermark);
}
+DAT_RETURN dat_extension(
+ IN DAT_HANDLE handle,
+ IN DAT_EXT_OP ext_op,
+ IN ... )
+
+{
+ DAT_RETURN status;
+ va_list args;
+
+ if (handle == NULL)
+ {
+ return DAT_ERROR(DAT_INVALID_HANDLE,
DAT_INVALID_HANDLE_EP);
+ }
+
+ /* verify provider extension support */
+ if (!dat_extensions)
+ {
+ return DAT_ERROR(DAT_NOT_IMPLEMENTED, 0);
+ }
+
+ va_start(args, ext_op);
+
+ status = DAT_EXTENSION(handle,
+ ext_op,
+ args);
+ va_end(args);
+
+ return status;
+}
+
/*
* Local variables:
* c-indent-level: 4
Index: dat/udat/udat.c
===================================================================
--- dat/udat/udat.c (revision 4594)
+++ dat/udat/udat.c (working copy)
@@ -66,6 +66,10 @@ udat_check_state ( void );
*
*
*********************************************************************/
+/*
+ * Use a global to get an unresolved when run with
pre-extension library
+ */
+int dat_extensions = 0;
/*
*
@@ -230,13 +234,44 @@ dat_ia_openv (
async_event_qlen,
async_event_handle,
ia_handle);
+
+ /*
+ * See if provider supports extensions
+ */
if (dat_status == DAT_SUCCESS)
{
- return_handle = dats_set_ia_handle (*ia_handle);
- if (return_handle >= 0)
- {
- *ia_handle = (DAT_IA_HANDLE)return_handle;
- }
+ DAT_PROVIDER_ATTR p_attr;
+ int i;
+
+ return_handle = dats_set_ia_handle (*ia_handle);
+ if (return_handle >= 0)
+ {
+ *ia_handle = (DAT_IA_HANDLE)return_handle;
+ }
+
+ if ( dat_ia_query( *ia_handle,
+ NULL,
+ 0,
+ NULL,
+
DAT_PROVIDER_FIELD_PROVIDER_SPECIFIC_ATTR,
+ &p_attr ) == DAT_SUCCESS )
+ {
+ for ( i = 0; i < p_attr.num_provider_specific_attr;
i++ )
+ {
+ if ( (strcmp(
p_attr.provider_specific_attr[i].name,
+
"DAT_EXTENSION_INTERFACE" ) == 0) &&
+ (strcmp(
p_attr.provider_specific_attr[i].value,
+ "TRUE" ) == 0) )
+ {
+
dat_os_dbg_print(DAT_OS_DBG_TYPE_CONSUMER_API,
+ "DAT Registry:
dat_ia_open () "
+ "DAPL Extension
Interface supported!\n");
+
+ dat_extensions = 1;
+ break;
+ }
+ }
+ }
}
return dat_status;
Index: README
===================================================================
--- README (revision 4589)
+++ README (working copy)
@@ -1,5 +1,10 @@
There are now 3 uDAPL providers for openib
(openib,openib_scm,openib_cma).
+NEW FEATURES for openib_cma provider:
+API extensions for immediate data and atomic operations have
been added.
+see dat/include/dat/dat_extensions.h for new API's.
+see dapl/test/dtest/dtest_ext.c for example test cast
+
==========
1.0 BUILD:
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20051223/a79f47b4/attachment.html>
More information about the general
mailing list