[openib-general] [RFC][PATCH] OpenIB uDAPL extension proposal - sample immed data and atomic api's
Arlin Davis
arlin.r.davis at intel.com
Thu Dec 22 15:20:29 PST 2005
James and Arkady,
DAPL provides a generalized abstraction to RDMA capable transports. As a generalized abstraction, it
cannot exploit the unique properties that many of the underlying platforms/interconnects can provide
so I would like to propose a simple (minimum impact on libdat) extensible interface to uDAPL that
will allow vendors to expose such capabilities. I am looking for feedback, especially from the DAT
collaborative. I have included both a design document and actual working code as a reference.
The patch provides a fully tested DAT and DAPL library (openib_cma) set with the following provider
extensions:
DAT_RETURN
dat_ep_post_write_immed(
IN DAT_EP_HANDLE ep_handle,
IN DAT_COUNT num_segments
IN DAT_LMR_TRIPLET *local_iov,
IN DAT_DTO_COOKIE user_cookie,
IN DAT_RMR_TRIPLE *remote_iov,
IN DAT_UINT32 immediate_data,
IN DAT_COMPLETION_FLAGS completion_flags);
DAT_RETURN
dat_ep_post_cmp_and_swap(
IN DAT_EP_HANDLE ep_handle,
IN DAT_UINT64 cmp_value,
IN DAT_UINT64 swap_value,
IN DAT_LMR_TRIPLE *local_iov,
IN DAT_DTO_COOKIE user_cookie,
IN DAT_RMR_TRIPLE *remote_iov,
IN DAT_COMPLETION_FLAGS completion_flags);
DAT_RETURN
dat_ep_post_fetch_and_add(
IN DAT_EP_HANDLE ep_handle,
IN DAT_UINT64 add_value,
IN DAT_LMR_TRIPLE *local_iov,
IN DAT_DTO_COOKIE user_cookie,
IN DAT_RMR_TRIPLE *remote_iov,
IN DAT_COMPLETION_FLAGS completion_flags);
Also, included is a sample program (dtest_ext.c) that can be used as a programming example.
Thanks,
-arlin
Signed-off by: Arlin Davis <mailto:ardavis at ichips.intel.com> <ardavis at ichips.intel.com>
Index: test/dtest/dat.conf
===================================================================
--- test/dtest/dat.conf (revision 4589)
+++ test/dtest/dat.conf (working copy)
@@ -1,11 +1,20 @@
#
-# DAT 1.1 and 1.2 configuration file
+# DAT 1.2 configuration file
#
# Each entry should have the following fields:
#
# <ia_name> <api_version> <threadsafety> <default> <lib_path> \
# <provider_version> <ia_params> <platform_params>
#
-# Example for openib using the first Mellanox adapter, port 1 and port 2
+# Example for openib_cma and openib_scm
+#
+# For scm version you specify <ia_params> as actual device name and port
+# For cma version you specify <ia_params> as:
+# network address, network hostname, or netdev name and 0 for port
+#
+OpenIB-scm1 u1.2 nonthreadsafe default /usr/local/openib_dapl/udapl/Target/libdapl.so mv_dapl.1.2
"mthca0 1" ""
+OpenIB-scm2 u1.2 nonthreadsafe default /usr/local/openib_dapl/udapl/Target/libdapl.so mv_dapl.1.2
"mthca0 2" ""
+OpenIB-cma-ip u1.2 nonthreadsafe default /usr/local/openib_dapl/udapl/Target/libdapl.so mv_dapl.1.2
"192.168.0.22 0" ""
+OpenIB-cma-name u1.2 nonthreadsafe default /usr/local/openib_dapl/udapl/Target/libdapl.so
mv_dapl.1.2 "svr1-ib0 0" ""
+OpenIB-cma-netdev u1.2 nonthreadsafe default /usr/local/openib_dapl/udapl/Target/libdapl.so
mv_dapl.1.2 "ib0 0" ""
-IB1 u1.2 nonthreadsafe default
Index: test/dtest/makefile
===================================================================
--- test/dtest/makefile (revision 4589)
+++ test/dtest/makefile (working copy)
@@ -4,13 +4,18 @@ CFLAGS = -O2 -g
DAT_INC = ../../dat/include
DAT_LIB = /usr/local/lib
-all: dtest
+all: dtest dtest_ext
clean:
- rm -f *.o;touch *.c;rm -f dtest
+ rm -f *.o;touch *.c;rm -f dtest dtest_ext
dtest: ./dtest.c
$(CC) $(CFLAGS) ./dtest.c -o dtest \
-DDAPL_PROVIDER='"OpenIB-cma-ip"' \
-I $(DAT_INC) -L $(DAT_LIB) -ldat
+dtest_ext: ./dtest_ext.c
+ $(CC) $(CFLAGS) ./dtest_ext.c -o dtest_ext \
+ -DDAPL_PROVIDER='"OpenIB-cma-ip"' \
+ -I $(DAT_INC) -L $(DAT_LIB) -ldat
+
Index: test/dtest/README
===================================================================
--- test/dtest/README (revision 4589)
+++ test/dtest/README (working copy)
@@ -1,10 +1,11 @@
simple dapl test just for initial openIB uDAPL testing...
dtest/dtest.c
+ dtest/dtest_ext.c
dtest/makefile
dtest/dat.conf
-to build (default uDAPL name == IB1, ib device == mthca0, port == 1)
+to build (default uDAPL name == OpenIB-cma-ip)
edit makefile and change path (DAT_LIB) to appropriate libdat.so
edit dat.conf and change path to appropriate libdapl.so
cp dat.conf to /etc/dat.conf
Index: dapl/include/dapl.h
===================================================================
--- dapl/include/dapl.h (revision 4589)
+++ dapl/include/dapl.h (working copy)
@@ -1,25 +1,28 @@
/*
- * Copyright (c) 2002-2003, Network Appliance, Inc. All rights reserved.
+ * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved.
*
* This Software is licensed under one of the following licenses:
- *
+ *
* 1) under the terms of the "Common Public License 1.0" a copy of which is
- * available from the Open Source Initiative, see
+ * in the file LICENSE.txt in the root directory. The license is also
+ * available from the Open Source Initiative, see
* http://www.opensource.org/licenses/cpl.php.
- *
- * 2) under the terms of the "The BSD License" a copy of which is
- * available from the Open Source Initiative, see
+ *
+ * 2) under the terms of the "The BSD License" a copy of which is in the file
+ * LICENSE2.txt in the root directory. The license is also available from
+ * the Open Source Initiative, see
* http://www.opensource.org/licenses/bsd-license.php.
- *
+ *
* 3) under the terms of the "GNU General Public License (GPL) Version 2" a
- * copy of which is available from the Open Source Initiative, see
+ * copy of which is in the file LICENSE3.txt in the root directory. The
+ * license is also available from the Open Source Initiative, see
* http://www.opensource.org/licenses/gpl-license.php.
- *
+ *
* Licensee has the right to choose one of the above licenses.
- *
+ *
* Redistributions of source code must retain the above copyright
* notice and one of the license notices.
- *
+ *
* Redistributions in binary form must reproduce both the above copyright
* notice, one of the license notices in the documentation
* and/or other materials provided with the distribution.
@@ -61,6 +64,8 @@
#include "dapl_dummy_util.h"
#elif OPENIB
#include "dapl_ib_util.h"
+#elif DET
+#include "dapl_det_util.h"
#endif
/*********************************************************************
@@ -213,6 +218,10 @@ typedef struct dapl_cookie DAPL_COOKIE;
typedef struct dapl_dto_cookie DAPL_DTO_COOKIE;
typedef struct dapl_rmr_cookie DAPL_RMR_COOKIE;
+#ifdef DAPL_EXTENSIONS
+typedef struct dapl_ext_cookie DAPL_EXT_COOKIE;
+#endif
+
typedef struct dapl_private DAPL_PRIVATE;
typedef void (*DAPL_CONNECTION_STATE_HANDLER) (
@@ -563,6 +572,13 @@ typedef enum dapl_dto_type
DAPL_DTO_TYPE_RECV,
DAPL_DTO_TYPE_RDMA_WRITE,
DAPL_DTO_TYPE_RDMA_READ,
+#ifdef DAPL_EXTENSIONS
+ DAPL_DTO_TYPE_RDMA_WRITE_IMMED,
+ DAPL_DTO_TYPE_RECV_IMMED,
+ DAPL_DTO_TYPE_CMP_AND_SWAP,
+ DAPL_DTO_TYPE_FETCH_AND_ADD,
+#endif
+
} DAPL_DTO_TYPE;
typedef enum dapl_cookie_type
@@ -570,6 +586,9 @@ typedef enum dapl_cookie_type
DAPL_COOKIE_TYPE_NULL,
DAPL_COOKIE_TYPE_DTO,
DAPL_COOKIE_TYPE_RMR,
+#ifdef DAPL_EXTENSIONS
+ DAPL_COOKIE_TYPE_EXTENSION,
+#endif
} DAPL_COOKIE_TYPE;
/* DAPL_DTO_COOKIE used as context for DTO WQEs */
@@ -587,6 +606,27 @@ struct dapl_rmr_cookie
DAT_RMR_COOKIE cookie;
};
+#ifdef DAPL_EXTENSIONS
+
+/* DAPL extended cookie types */
+typedef enum dapl_ext_type
+{
+ DAPL_EXT_TYPE_RDMA_WRITE_IMMED,
+ DAPL_EXT_TYPE_CMP_AND_SWAP,
+ DAPL_EXT_TYPE_FETCH_AND_ADD,
+ DAPL_EXT_TYPE_RECV
+} DAPL_EXT_TYPE;
+
+/* DAPL extended cookie */
+struct dapl_ext_cookie
+{
+ DAPL_EXT_TYPE type;
+ DAT_DTO_COOKIE cookie;
+ DAT_COUNT size; /* used RDMA write with immed */
+};
+
+#endif
+
/* DAPL_COOKIE used as context for WQEs */
struct dapl_cookie
{
@@ -597,6 +637,9 @@ struct dapl_cookie
{
DAPL_DTO_COOKIE dto;
DAPL_RMR_COOKIE rmr;
+#ifdef DAPL_EXTENSIONS
+ DAPL_EXT_COOKIE ext;
+#endif
} val;
};
@@ -1116,6 +1159,15 @@ extern DAT_RETURN dapl_srq_set_lw(
IN DAT_SRQ_HANDLE, /* srq_handle */
IN DAT_COUNT); /* low_watermark */
+#ifdef DAPL_EXTENSIONS
+
+extern DAT_RETURN dapl_extensions(
+ IN DAT_HANDLE, /* dat_handle */
+ IN DAT_EXT_OP, /* extension operation */
+ IN va_list ); /* va_list args */
+
+#endif
+
/*
* DAPL internal utility function prototpyes
*/
Index: dapl/udapl/Makefile
===================================================================
--- dapl/udapl/Makefile (revision 4589)
+++ dapl/udapl/Makefile (working copy)
@@ -156,6 +156,7 @@ PROVIDER = $(TOPDIR)/../openib_cma
CFLAGS += -DOPENIB
CFLAGS += -DCQ_WAIT_OBJECT
CFLAGS += -I/usr/local/include/infiniband
+CFLAGS += -I/usr/local/include/rdma
endif
#
@@ -168,6 +169,12 @@ endif
# VN_MEM_SHARED_VIRTUAL_SUPPORT
# CFLAGS += -DVN_MEM_SHARED_VIRTUAL_SUPPORT=1
+# If an implementation supports DAPL extensions
+CFLAGS += -DDAPL_EXTENSIONS
+
+# If an implementation supports DAPL provider specific attributes
+CFLAGS += -DDAPL_PROVIDER_SPECIFIC_ATTR
+
CFLAGS += -I.
CFLAGS += -I..
CFLAGS += -I../../dat/include
@@ -283,6 +290,8 @@ LDFLAGS += -libverbs -lrdmacm
LDFLAGS += -rpath /usr/local/lib -L /usr/local/lib
PROVIDER_SRCS = dapl_ib_util.c dapl_ib_cq.c dapl_ib_qp.c \
dapl_ib_cm.c dapl_ib_mem.c
+# implementation supports DAPL extensions
+PROVIDER_SRCS += dapl_ib_extensions.c
endif
UDAPL_SRCS = dapl_init.c \
Index: dapl/common/dapl_ia_query.c
===================================================================
--- dapl/common/dapl_ia_query.c (revision 4589)
+++ dapl/common/dapl_ia_query.c (working copy)
@@ -167,6 +167,14 @@ dapl_ia_query (
#if !defined(__KDAPL__)
provider_attr->pz_support = DAT_PZ_UNIQUE;
#endif /* !KDAPL */
+
+ /*
+ * Have provider set their own.
+ */
+#ifdef DAPL_PROVIDER_SPECIFIC_ATTR
+ dapls_set_provider_specific_attr(provider_attr);
+#endif
+
/*
* Set up evd_stream_merging_supported options. Note there is
* one bit per allowable combination, using the ordinal
Index: dapl/common/dapl_adapter_util.h
===================================================================
--- dapl/common/dapl_adapter_util.h (revision 4589)
+++ dapl/common/dapl_adapter_util.h (working copy)
@@ -256,6 +256,21 @@ dapls_ib_wait_object_wait (
IN u_int32_t timeout);
#endif
+#ifdef DAPL_PROVIDER_SPECIFIC_ATTR
+void
+dapls_set_provider_specific_attr(
+ IN DAT_PROVIDER_ATTR *provider_attr );
+#endif
+
+#ifdef DAPL_EXTENSIONS
+void
+dapls_cqe_to_event_extension(
+ IN DAPL_EP *ep_ptr,
+ IN DAPL_COOKIE *cookie,
+ IN ib_work_completion_t *cqe_ptr,
+ OUT DAT_EVENT *event_ptr);
+#endif
+
/*
* Values for provider DAT_NAMED_ATTR
*/
@@ -272,6 +287,8 @@ dapls_ib_wait_object_wait (
#include "dapl_dummy_dto.h"
#elif OPENIB
#include "dapl_ib_dto.h"
+#elif DET
+#include "dapl_det_dto.h"
#endif
Index: dapl/common/dapl_provider.c
===================================================================
--- dapl/common/dapl_provider.c (revision 4589)
+++ dapl/common/dapl_provider.c (working copy)
@@ -221,7 +221,11 @@ DAT_PROVIDER g_dapl_provider_template =
&dapl_srq_post_recv,
&dapl_srq_query,
&dapl_srq_resize,
- &dapl_srq_set_lw
+ &dapl_srq_set_lw,
+
+#ifdef DAPL_EXTENSIONS
+ &dapl_extensions
+#endif
};
#endif /* __KDAPL__ */
Index: dapl/common/dapl_evd_util.c
===================================================================
--- dapl/common/dapl_evd_util.c (revision 4589)
+++ dapl/common/dapl_evd_util.c (working copy)
@@ -502,6 +502,20 @@ dapli_evd_eh_print_cqe (
#ifdef DAPL_DBG
static char *optable[] =
{
+#ifdef OPENIB
+ /* different order for openib verbs */
+ "OP_RDMA_WRITE",
+ "OP_RDMA_WRITE_IMM",
+ "OP_SEND",
+ "OP_SEND_IMM",
+ "OP_RDMA_READ",
+ "OP_COMP_AND_SWAP",
+ "OP_FETCH_AND_ADD",
+ "OP_RECEIVE",
+ "OP_RECEIVE_IMM",
+ "OP_BIND_MW",
+ "OP_INVALID",
+#else
"OP_SEND",
"OP_RDMA_READ",
"OP_RDMA_WRITE",
@@ -509,6 +523,7 @@ dapli_evd_eh_print_cqe (
"OP_FETCH_AND_ADD",
"OP_RECEIVE",
"OP_BIND_MW",
+#endif
0
};
@@ -1113,6 +1128,15 @@ dapli_evd_cqe_to_event (
dapls_cookie_dealloc (&ep_ptr->req_buffer, cookie);
break;
}
+
+#ifdef DAPL_EXTENSIONS
+ case DAPL_COOKIE_TYPE_EXTENSION:
+ {
+ dapls_cqe_to_event_extension(ep_ptr, cookie, cqe_ptr, event_ptr);
+ break;
+ }
+#endif
+
default:
{
dapl_os_assert (!"Invalid Operation type");
Index: dapl/openib_cma/dapl_ib_dto.h
===================================================================
--- dapl/openib_cma/dapl_ib_dto.h (revision 4589)
+++ dapl/openib_cma/dapl_ib_dto.h (working copy)
@@ -35,7 +35,7 @@
*
* Description:
*
- * The uDAPL openib provider - DTO operations and CQE macros
+ * The OpenIB uCMA provider - DTO operations and CQE macros
*
****************************************************************************
* Source Control System Information
@@ -119,7 +119,6 @@ dapls_ib_post_recv (
return DAT_SUCCESS;
}
-
/*
* dapls_ib_post_send
*
@@ -191,7 +190,7 @@ dapls_ib_post_send (
if (cookie != NULL)
cookie->val.dto.size = total_len;
-
+
if ((op_type == OP_RDMA_WRITE) || (op_type == OP_RDMA_READ)) {
wr.wr.rdma.remote_addr = remote_iov->target_address;
wr.wr.rdma.rkey = remote_iov->rmr_context;
@@ -224,6 +223,152 @@ dapls_ib_post_send (
return DAT_SUCCESS;
}
+#ifdef DAPL_EXTENSIONS
+/*
+ * dapls_ib_post_ext_send
+ *
+ * Provider specific extended Post SEND function
+ */
+STATIC _INLINE_ DAT_RETURN
+dapls_ib_post_ext_send (
+ IN DAPL_EP *ep_ptr,
+ IN ib_send_op_type_t op_type,
+ IN DAPL_COOKIE *cookie,
+ IN DAT_COUNT segments,
+ IN DAT_LMR_TRIPLET *local_iov,
+ IN const DAT_RMR_TRIPLET *remote_iov,
+ IN DAT_UINT32 idata,
+ IN DAT_UINT64 compare_add,
+ IN DAT_UINT64 swap,
+ IN DAT_COMPLETION_FLAGS completion_flags)
+{
+ dapl_dbg_log(DAPL_DBG_TYPE_EP,
+ " post_snd: ep %p op %d ck %p sgs",
+ "%d l_iov %p r_iov %p f %d\n",
+ ep_ptr, op_type, cookie, segments, local_iov,
+ remote_iov, completion_flags);
+
+ ib_data_segment_t ds_array[DEFAULT_DS_ENTRIES];
+ ib_data_segment_t *ds_array_p;
+ struct ibv_send_wr wr;
+ struct ibv_send_wr *bad_wr;
+ ib_hca_transport_t *ibt_ptr =
+ &ep_ptr->header.owner_ia->hca_ptr->ib_trans;
+ DAT_COUNT i, total_len;
+
+ dapl_dbg_log(DAPL_DBG_TYPE_EP,
+ " post_snd: ep %p cookie %p segs %d l_iov %p\n",
+ ep_ptr, cookie, segments, local_iov);
+
+ if(segments <= DEFAULT_DS_ENTRIES)
+ ds_array_p = ds_array;
+ else
+ ds_array_p =
+ dapl_os_alloc(segments * sizeof(ib_data_segment_t));
+
+ if (NULL == ds_array_p)
+ return (DAT_INSUFFICIENT_RESOURCES);
+
+ /* setup the work request */
+ wr.next = 0;
+ wr.opcode = op_type;
+ wr.num_sge = 0;
+ wr.send_flags = 0;
+ wr.wr_id = (uint64_t)(uintptr_t)cookie;
+ wr.sg_list = ds_array_p;
+ total_len = 0;
+
+ for (i = 0; i < segments; i++ ) {
+ if ( !local_iov[i].segment_length )
+ continue;
+
+ ds_array_p->addr = (uint64_t) local_iov[i].virtual_address;
+ ds_array_p->length = local_iov[i].segment_length;
+ ds_array_p->lkey = local_iov[i].lmr_context;
+
+ dapl_dbg_log(DAPL_DBG_TYPE_EP,
+ " post_snd: lkey 0x%x va %p len %d\n",
+ ds_array_p->lkey, ds_array_p->addr,
+ ds_array_p->length );
+
+ total_len += ds_array_p->length;
+ wr.num_sge++;
+ ds_array_p++;
+ }
+
+ if (cookie != NULL)
+ cookie->val.dto.size = total_len;
+
+ if ((op_type == OP_RDMA_WRITE) ||
+ (op_type == OP_RDMA_WRITE_IMM) ||
+ (op_type == OP_RDMA_READ)) {
+ wr.wr.rdma.remote_addr = remote_iov->target_address;
+ wr.wr.rdma.rkey = remote_iov->rmr_context;
+ dapl_dbg_log(DAPL_DBG_TYPE_EP,
+ " post_snd_rdma: rkey 0x%x va %#016Lx\n",
+ wr.wr.rdma.rkey, wr.wr.rdma.remote_addr);
+ }
+
+ switch (op_type) {
+ case OP_RDMA_WRITE_IMM:
+ dapl_dbg_log(DAPL_DBG_TYPE_EP,
+ " post_snd: OP_RDMA_WRITE_IMMED=0x%x\n", idata );
+ wr.imm_data = idata;
+ break;
+ case OP_COMP_AND_SWAP:
+ /* OP_COMP_AND_SWAP has direct IBAL wr_type mapping */
+ dapl_dbg_log(DAPL_DBG_TYPE_EP,
+ " post_snd: OP_COMP_AND_SWAP=%lx,"
+ "%lx rkey 0x%x va %#016Lx\n",
+ compare_add, swap, remote_iov->rmr_context,
+ remote_iov->target_address);
+
+ wr.wr.atomic.compare_add = compare_add;
+ wr.wr.atomic.swap = swap;
+ wr.wr.atomic.remote_addr = remote_iov->target_address;
+ wr.wr.atomic.rkey = remote_iov->rmr_context;
+ break;
+ case OP_FETCH_AND_ADD:
+ /* OP_FETCH_AND_ADD has direct IBAL wr_type mapping */
+ dapl_dbg_log(DAPL_DBG_TYPE_EP,
+ " post_snd: OP_FETCH_AND_ADD=%lx,"
+ "%lx rkey 0x%x va %#016Lx\n",
+ compare_add, remote_iov->rmr_context,
+ remote_iov->target_address);
+
+ wr.wr.atomic.compare_add = compare_add;
+ wr.wr.atomic.remote_addr = remote_iov->target_address;
+ wr.wr.atomic.rkey = remote_iov->rmr_context;
+ break;
+ default:
+ break;
+ }
+
+ /* inline data for send or write ops */
+ if ((total_len <= ibt_ptr->max_inline_send) &&
+ ((op_type == OP_SEND) || (op_type == OP_RDMA_WRITE)))
+ wr.send_flags |= IBV_SEND_INLINE;
+
+ /* set completion flags in work request */
+ wr.send_flags |= (DAT_COMPLETION_SUPPRESS_FLAG &
+ completion_flags) ? 0 : IBV_SEND_SIGNALED;
+ wr.send_flags |= (DAT_COMPLETION_BARRIER_FENCE_FLAG &
+ completion_flags) ? IBV_SEND_FENCE : 0;
+ wr.send_flags |= (DAT_COMPLETION_SOLICITED_WAIT_FLAG &
+ completion_flags) ? IBV_SEND_SOLICITED : 0;
+
+ dapl_dbg_log(DAPL_DBG_TYPE_EP,
+ " post_snd: op 0x%x flags 0x%x sglist %p, %d\n",
+ wr.opcode, wr.send_flags, wr.sg_list, wr.num_sge);
+
+ if (ibv_post_send(ep_ptr->qp_handle->cm_id->qp, &wr, &bad_wr))
+ return( dapl_convert_errno(EFAULT,"ibv_recv") );
+
+ dapl_dbg_log(DAPL_DBG_TYPE_EP," post_snd: returned\n");
+ return DAT_SUCCESS;
+}
+#endif
+
STATIC _INLINE_ DAT_RETURN
dapls_ib_optional_prv_dat(
IN DAPL_CR *cr_ptr,
Index: dapl/openib_cma/dapl_ib_util.c
===================================================================
--- dapl/openib_cma/dapl_ib_util.c (revision 4589)
+++ dapl/openib_cma/dapl_ib_util.c (working copy)
@@ -35,7 +35,7 @@
*
* Description:
*
- * The uDAPL openib provider - init, open, close, utilities, work thread
+ * The OpenIB uCMA provider - init, open, close, utilities, work thread
*
****************************************************************************
* Source Control System Information
@@ -64,7 +64,6 @@ static const char rcsid[] = "$Id: $";
#include <net/if.h> /* for struct ifreq */
#include <net/if_arp.h> /* for ARPHRD_INFINIBAND */
-
int g_dapl_loopback_connection = 0;
int g_ib_pipe[2];
ib_thread_state_t g_ib_thread_state = 0;
@@ -727,7 +726,7 @@ void dapli_thread(void *arg)
int ret,idx,fds;
char rbuf[2];
- dapl_dbg_log (DAPL_DBG_TYPE_CM,
+ dapl_dbg_log (DAPL_DBG_TYPE_UTIL,
" ib_thread(%d,0x%x): ENTER: pipe %d ucma %d\n",
getpid(), g_ib_thread, g_ib_pipe[0], rdma_get_fd());
@@ -767,7 +766,7 @@ void dapli_thread(void *arg)
ufds[idx].revents = 0;
uhca[idx] = hca;
- dapl_dbg_log(DAPL_DBG_TYPE_CM,
+ dapl_dbg_log(DAPL_DBG_TYPE_UTIL,
" ib_thread(%d) poll_fd: hca[%d]=%p, async=%d"
" pipe=%d cm=%d cq=d\n",
getpid(), hca, ufds[idx-1].fd,
@@ -783,14 +782,14 @@ void dapli_thread(void *arg)
dapl_os_unlock(&g_hca_lock);
ret = poll(ufds, fds, -1);
if (ret <= 0) {
- dapl_dbg_log(DAPL_DBG_TYPE_WARN,
+ dapl_dbg_log(DAPL_DBG_TYPE_UTIL,
" ib_thread(%d): ERR %s poll\n",
getpid(),strerror(errno));
dapl_os_lock(&g_hca_lock);
continue;
}
- dapl_dbg_log(DAPL_DBG_TYPE_CM,
+ dapl_dbg_log(DAPL_DBG_TYPE_UTIL,
" ib_thread(%d) poll_event: "
" async=0x%x pipe=0x%x cm=0x%x cq=0x%x\n",
getpid(), ufds[idx-1].revents, ufds[0].revents,
@@ -834,3 +833,63 @@ void dapli_thread(void *arg)
dapl_os_unlock(&g_hca_lock);
}
+#ifdef DAPL_PROVIDER_SPECIFIC_ATTR
+/*
+ * dapls_set_provider_specific_attr
+ *
+ * Input:
+ * attr_ptr Pointer provider attributes
+ *
+ * Output:
+ * none
+ *
+ * Returns:
+ * void
+ */
+DAT_NAMED_ATTR ib_attrs[] = {
+
+#ifdef DAPL_EXTENSIONS
+ {
+ DAT_EXT_ATTR,
+ DAT_EXT_ATTR_TRUE
+ },
+ {
+ DAT_EXT_ATTR_RDMA_WRITE_IMMED,
+ DAT_EXT_ATTR_TRUE
+ },
+ {
+ DAT_EXT_ATTR_RECV_IMMED,
+ DAT_EXT_ATTR_TRUE
+ },
+ /* inbound immediate data placed in event, NOT payload */
+ {
+ DAT_EXT_ATTR_RECV_IMMED_EVENT,
+ DAT_EXT_ATTR_TRUE
+ },
+ {
+ DAT_EXT_ATTR_FETCH_AND_ADD,
+ DAT_EXT_ATTR_TRUE
+ },
+ {
+ DAT_EXT_ATTR_CMP_AND_SWAP,
+ DAT_EXT_ATTR_TRUE
+ },
+#else
+ {
+ "DAT_EXTENSION_INTERFACE",
+ "FALSE"
+ },
+#endif
+};
+
+#define SPEC_ATTR_SIZE(x) ( sizeof(x)/sizeof(DAT_NAMED_ATTR) )
+
+void dapls_set_provider_specific_attr(
+ IN DAT_PROVIDER_ATTR *attr_ptr )
+{
+ attr_ptr->num_provider_specific_attr = SPEC_ATTR_SIZE(ib_attrs);
+ attr_ptr->provider_specific_attr = ib_attrs;
+}
+
+#endif
+
Index: dapl/openib_cma/dapl_ib_mem.c
===================================================================
--- dapl/openib_cma/dapl_ib_mem.c (revision 4589)
+++ dapl/openib_cma/dapl_ib_mem.c (working copy)
@@ -25,9 +25,9 @@
/**********************************************************************
*
- * MODULE: dapl_det_mem.c
+ * MODULE: dapl_ib_mem.c
*
- * PURPOSE: Intel DET APIs: Memory windows, registration,
+ * PURPOSE: OpenIB uCMA provider Memory windows, registration,
* and protection domain
*
* $Id: $
@@ -72,12 +72,10 @@ dapls_convert_privileges(IN DAT_MEM_PRIV
access |= IBV_ACCESS_LOCAL_WRITE;
if (DAT_MEM_PRIV_REMOTE_WRITE_FLAG & privileges)
access |= IBV_ACCESS_REMOTE_WRITE;
- if (DAT_MEM_PRIV_REMOTE_READ_FLAG & privileges)
- access |= IBV_ACCESS_REMOTE_READ;
- if (DAT_MEM_PRIV_REMOTE_READ_FLAG & privileges)
- access |= IBV_ACCESS_REMOTE_READ;
- if (DAT_MEM_PRIV_REMOTE_READ_FLAG & privileges)
+ if (DAT_MEM_PRIV_REMOTE_READ_FLAG & privileges) {
access |= IBV_ACCESS_REMOTE_READ;
+ access |= IBV_ACCESS_REMOTE_ATOMIC;
+ }
return access;
}
Index: dapl/openib_cma/dapl_ib_cm.c
===================================================================
--- dapl/openib_cma/dapl_ib_cm.c (revision 4589)
+++ dapl/openib_cma/dapl_ib_cm.c (working copy)
@@ -35,7 +35,7 @@
*
* Description:
*
- * The uDAPL openib provider - connection management
+ * The OpenIB uCMA provider - uCMA connection management
*
****************************************************************************
* Source Control System Information
@@ -592,7 +592,11 @@ dapls_ib_setup_conn_listener(IN DAPL_IA
if (rdma_bind_addr(conn->cm_id,
(struct sockaddr *)&ia_ptr->hca_ptr->hca_address)) {
- dat_status = dapl_convert_errno(errno,"setup_listener");
+ if (errno == -EBUSY)
+ dat_status = DAT_CONN_QUAL_IN_USE;
+ else
+ dat_status =
+ dapl_convert_errno(errno,"setup_listener");
goto bail;
}
Index: dapl/openib_cma/dapl_ib_qp.c
===================================================================
--- dapl/openib_cma/dapl_ib_qp.c (revision 4589)
+++ dapl/openib_cma/dapl_ib_qp.c (working copy)
@@ -25,9 +25,9 @@
/**********************************************************************
*
- * MODULE: dapl_det_qp.c
+ * MODULE: dapl_ib_qp.c
*
- * PURPOSE: QP routines for access to DET Verbs
+ * PURPOSE: OpenIB uCMA QP routines
*
* $Id: $
**********************************************************************/
Index: dapl/openib_cma/dapl_ib_util.h
===================================================================
--- dapl/openib_cma/dapl_ib_util.h (revision 4589)
+++ dapl/openib_cma/dapl_ib_util.h (working copy)
@@ -35,7 +35,7 @@
*
* Description:
*
- * The uDAPL openib provider - definitions, prototypes,
+ * The OpenIB uCMA provider - definitions, prototypes,
*
****************************************************************************
* Source Control System Information
Index: dapl/openib_cma/README
===================================================================
--- dapl/openib_cma/README (revision 4589)
+++ dapl/openib_cma/README (working copy)
@@ -23,15 +23,22 @@ New files for openib_scm provider
dapl/openib_cma/dapl_ib_util.c
dapl/openib_cma/dapl_ib_util.h
dapl/openib_cma/dapl_ib_cm.c
+ dapl/openib_cma/dapl_ib_extensions.c
A simple dapl test just for openib_scm testing...
test/dtest/dtest.c
+ test/dtest/dtest_ext.c
test/dtest/makefile
server: dtest -s
client: dtest -h hostname
+or with extensions
+
+ server: dtest_ext -s
+ client: dtest_ext -h hostname
+
known issues:
no memory windows support in ibverbs, dat_create_rmr fails.
Index: dapl/openib_cma/dapl_ib_cq.c
===================================================================
--- dapl/openib_cma/dapl_ib_cq.c (revision 4589)
+++ dapl/openib_cma/dapl_ib_cq.c (working copy)
@@ -35,7 +35,7 @@
*
* Description:
*
- * The uDAPL openib provider - completion queue
+ * The OpenIB uCMA provider - completion queue
*
****************************************************************************
* Source Control System Information
@@ -498,7 +498,10 @@ dapls_ib_wait_object_wait(IN ib_wait_obj
if (timeout != DAT_TIMEOUT_INFINITE)
timeout_ms = timeout/1000;
- status = poll(&cq_fd, 1, timeout_ms);
+ /* restart syscall */
+ while ((status = poll(&cq_fd, 1, timeout_ms)) == -1 )
+ if (errno == EINTR)
+ continue;
/* returned event */
if (status > 0) {
@@ -511,6 +514,8 @@ dapls_ib_wait_object_wait(IN ib_wait_obj
/* timeout */
} else if (status == 0)
status = ETIMEDOUT;
+ else
+ status = errno;
dapl_dbg_log(DAPL_DBG_TYPE_UTIL,
" cq_object_wait: RET evd %p ibv_cq %p ibv_ctx %p %s\n",
Index: dat/include/dat/dat_redirection.h
===================================================================
--- dat/include/dat/dat_redirection.h (revision 4589)
+++ dat/include/dat/dat_redirection.h (working copy)
@@ -59,10 +59,10 @@ typedef struct dat_provider DAT_PROVIDER
* This would allow a good compiler to avoid indirection overhead when
* making function calls.
*/
-
#define DAT_HANDLE_TO_PROVIDER(handle) (*(DAT_PROVIDER **)(handle))
#endif
+
#define DAT_IA_QUERY(ia, evd, ia_msk, ia_ptr, p_msk, p_ptr) \
(*DAT_HANDLE_TO_PROVIDER (ia)->ia_query_func) (\
(ia), \
@@ -395,6 +395,12 @@ typedef struct dat_provider DAT_PROVIDER
(lbuf), \
(cookie))
+#define DAT_EXTENSION(handle, op, args) \
+ (*DAT_HANDLE_TO_PROVIDER (handle)->extension_func) (\
+ (handle), \
+ (op), \
+ (args))
+
/***************************************************************
*
* FUNCTION PROTOTYPES
@@ -720,4 +726,11 @@ typedef DAT_RETURN (*DAT_SRQ_POST_RECV_F
IN DAT_LMR_TRIPLET *, /* local_iov */
IN DAT_DTO_COOKIE ); /* user_cookie */
+/* Extension function */
+#include <stdarg.h>
+typedef DAT_RETURN (*DAT_EXTENSION_FUNC) (
+ IN DAT_HANDLE, /* dat handle */
+ IN DAT_EXT_OP, /* extension operation */
+ IN va_list ); /* va_list */
+
#endif /* _DAT_REDIRECTION_H_ */
Index: dat/include/dat/dat.h
===================================================================
--- dat/include/dat/dat.h (revision 4589)
+++ dat/include/dat/dat.h (working copy)
@@ -854,11 +854,15 @@ typedef enum dat_event_number
DAT_ASYNC_ERROR_EP_BROKEN = 0x08003,
DAT_ASYNC_ERROR_TIMED_OUT = 0x08004,
DAT_ASYNC_ERROR_PROVIDER_INTERNAL_ERROR = 0x08005,
- DAT_SOFTWARE_EVENT = 0x10001
+ DAT_SOFTWARE_EVENT = 0x10001,
+ DAT_EXTENSION_EVENT = 0x20001
+
} DAT_EVENT_NUMBER;
-/* Union for event Data */
+/* include extension data definitions */
+#include <dat/dat_extensions.h>
+/* Union for event Data */
typedef union dat_event_data
{
DAT_DTO_COMPLETION_EVENT_DATA dto_completion_event_data;
@@ -867,6 +871,7 @@ typedef union dat_event_data
DAT_CONNECTION_EVENT_DATA connect_event_data;
DAT_ASYNCH_ERROR_EVENT_DATA asynch_error_event_data;
DAT_SOFTWARE_EVENT_DATA software_event_data;
+ DAT_EXTENSION_DATA extension_data;
} DAT_EVENT_DATA;
/* Event struct that holds all event information */
@@ -1222,6 +1227,11 @@ extern DAT_RETURN dat_srq_set_lw (
IN DAT_SRQ_HANDLE, /* srq_handle */
IN DAT_COUNT); /* low_watermark */
+extern DAT_RETURN dat_extension(
+ IN DAT_HANDLE,
+ IN DAT_EXT_OP,
+ IN ... );
+
/*
* DAT registry functions.
*
Index: dat/include/dat/udat_redirection.h
===================================================================
--- dat/include/dat/udat_redirection.h (revision 4589)
+++ dat/include/dat/udat_redirection.h (working copy)
@@ -199,7 +199,6 @@ typedef DAT_RETURN (*DAT_EVD_SET_UNWAITA
typedef DAT_RETURN (*DAT_EVD_CLEAR_UNWAITABLE_FUNC) (
IN DAT_EVD_HANDLE); /* evd_handle */
-
#include <dat/dat_redirection.h>
struct dat_provider
@@ -294,6 +293,10 @@ struct dat_provider
DAT_SRQ_QUERY_FUNC srq_query_func;
DAT_SRQ_RESIZE_FUNC srq_resize_func;
DAT_SRQ_SET_LW_FUNC srq_set_lw_func;
+
+ /* extension for provder specific functions */
+ DAT_EXTENSION_FUNC extension_func;
+
};
#endif /* _UDAT_REDIRECTION_H_ */
Index: dat/include/dat/dat_extensions.h
===================================================================
--- dat/include/dat/dat_extensions.h (revision 0)
+++ dat/include/dat/dat_extensions.h (revision 0)
@@ -0,0 +1,209 @@
+/*
+ * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved.
+ *
+ * This Software is licensed under one of the following licenses:
+ *
+ * 1) under the terms of the "Common Public License 1.0" a copy of which is
+ * in the file LICENSE.txt in the root directory. The license is also
+ * available from the Open Source Initiative, see
+ * http://www.opensource.org/licenses/cpl.php.
+ *
+ * 2) under the terms of the "The BSD License" a copy of which is in the file
+ * LICENSE2.txt in the root directory. The license is also available from
+ * the Open Source Initiative, see
+ * http://www.opensource.org/licenses/bsd-license.php.
+ *
+ * 3) under the terms of the "GNU General Public License (GPL) Version 2" a
+ * copy of which is in the file LICENSE3.txt in the root directory. The
+ * license is also available from the Open Source Initiative, see
+ * http://www.opensource.org/licenses/gpl-license.php.
+ *
+ * Licensee has the right to choose one of the above licenses.
+ *
+ * Redistributions of source code must retain the above copyright
+ * notice and one of the license notices.
+ *
+ * Redistributions in binary form must reproduce both the above copyright
+ * notice, one of the license notices in the documentation
+ * and/or other materials provided with the distribution.
+ */
+/**********************************************************************
+ *
+ * HEADER: dat_extensions.h
+ *
+ * PURPOSE: defines the extensions to the DAT API for uDAPL.
+ *
+ * Description: Header file for "uDAPL: User Direct Access Programming
+ * Library, Version: 1.2"
+ *
+ * Mapping rules:
+ * All global symbols are prepended with "DAT_" or "dat_"
+ * All DAT objects have an 'api' tag which, such as 'ep' or 'lmr'
+ * The method table is in the provider definition structure.
+ *
+ *
+ **********************************************************************/
+
+#ifndef _DAT_EXTENSIONS_H_
+
+extern int dat_extensions;
+
+/*
+ * Provider specific attribute strings for extension support
+ * returned with dat_ia_query() and
+ * DAT_PROVIDER_ATTR_MASK == DAT_PROVIDER_FIELD_PROVIDER_SPECIFIC_ATTR
+ *
+ * DAT_NAMED_ATTR name == extended operation,
+ * value == TRUE if extended operation is supported
+ */
+#define DAT_EXT_ATTR "DAT_EXTENSION_INTERFACE"
+#define DAT_EXT_ATTR_RDMA_WRITE_IMMED "DAT_EXT_RDMA_WRITE_IMMED"
+#define DAT_EXT_ATTR_RECV_IMMED "DAT_EXT_RECV_IMMED"
+#define DAT_EXT_ATTR_RECV_IMMED_EVENT "DAT_EXT_RECV_IMMED_EVENT"
+#define DAT_EXT_ATTR_RECV_IMMED_PAYLOAD "DAT_EXT_RECV_IMMED_PAYLOAD"
+#define DAT_EXT_ATTR_FETCH_AND_ADD "DAT_EXT_FETCH_AND_ADD"
+#define DAT_EXT_ATTR_CMP_AND_SWAP "DAT_EXT_CMP_AND_SWAP"
+#define DAT_EXT_ATTR_TRUE "TRUE"
+#define DAT_EXT_ATTR_FALSE "FALSE"
+
+/*
+ * Extension OPERATIONS
+ */
+typedef enum dat_ext_op
+{
+ DAT_EXT_RDMA_WRITE_IMMED,
+ DAT_EXT_RECV_IMMED,
+ DAT_EXT_FETCH_AND_ADD,
+ DAT_EXT_CMP_AND_SWAP,
+
+} DAT_EXT_OP;
+
+/*
+ * Extension completion event TYPES
+ */
+typedef enum dat_ext_event_type
+{
+ DAT_EXT_RDMA_WRITE_IMMED_STATUS = 1,
+ DAT_EXT_RECV_NO_IMMED,
+ DAT_EXT_RECV_IMMED_DATA_EVENT,
+ DAT_EXT_RECV_IMMED_DATA_PAYLOAD,
+ DAT_EXT_FETCH_AND_ADD_STATUS,
+ DAT_EXT_CMP_AND_SWAP_STATUS,
+
+} DAT_EXT_EVENT_TYPE;
+
+/*
+ * Extension completion event DATA
+ */
+typedef struct dat_immediate_data
+{
+ DAT_UINT32 data;
+
+} DAT_RDMA_WRITE_IMMED_DATA;
+
+typedef struct dat_extension_data
+{
+ DAT_DTO_COMPLETION_EVENT_DATA dto;
+ DAT_EXT_EVENT_TYPE type;
+ union {
+ DAT_RDMA_WRITE_IMMED_DATA immed;
+ } val;
+} DAT_EXTENSION_DATA;
+
+typedef enum dat_ext_flags
+{
+ DAT_EXT_WRITE_IMMED_FLAG = 0x1,
+ DAT_EXT_WRITE_CONFIRM_FLAG = 0x2,
+
+} DAT_EXT_FLAGS;
+
+/*
+ * Extended API with redirection via DAT extension function
+ */
+
+/*
+ * RDMA Write with IMMEDIATE extension:
+ *
+ * Asynchronous call performs a normal RDMA write to the remote endpoint
+ * followed by a post of an extended immediate data value to the receive
+ * EVD on the remote endpoint. Event completion for the request completes
+ * as an DAT_EXTENSION_EVENT with type set to DAT_EXT_RDMA_WRITE_IMMED_STATUS.
+ * Event completion on the remote endpoint completes as an DAT_EXTENSION_EVENT
+ * with type set to DAT_EXT_RECV_IMMED_DATA_IN_EVENT or
+ * DAT_EXT_RECV_IMMED_DATA_IN_PAYLOAD depending on the provider transport.
+ *
+ * DAT_EXT_WRITE_IMMED_FLAG requests that the supplied
+ *'immediate' value be sent as the payload of a four byte send following
+ * the RDMA Write, or any transport-dependent equivalent thereof.
+ * For example, on InfiniBand the request should be translated as an
+ * RDMA Write with Immediate.
+ *
+ * DAT_EXT_WRITE_CONFIRM_FLAG requests that this DTO
+ * not complete until receipt by the far end is confirmed.
+ *
+ * Note to Consumers: the immediate data will consume a receive
+ * buffer at the Data Sink.
+ *
+ * Other extension flags:
+ * n/a
+ */
+#define dat_ep_post_rdma_write_immed(ep, size, lbuf, cookie, rbuf, idata, eflgs, flgs) \
+ dat_extension( ep, \
+ DAT_EXT_RDMA_WRITE_IMMED, \
+ (size), \
+ (lbuf), \
+ (cookie), \
+ (rbuf), \
+ (idata), \
+ (eflgs), \
+ (flgs))
+
+/*
+ * Call performs a normal post receive message to the local endpoint
+ * that includes additional 32-bit buffer space for immediate data
+ * Event completion for the request completes as an
+ * DAT_EXTENSION_EVENT with type set to DAT_EXT_RDMA_WRITE_IMMED_STATUS.
+ */
+#define dat_ep_post_recv_immed(ep, size, lbuf, cookie, flgs) \
+ dat_extension( ep, \
+ DAT_EXT_RECV_IMMED, \
+ (size), \
+ (lbuf), \
+ (cookie), \
+ (flgs))
+
+/*
+ * This asynchronous call is modeled after the InfiniBand atomic
+ * Fetch and Add operation. The add_value is added to the 64 bit
+ * value stored at the remote memory location specified in remote_iov
+ * and the result is stored in the local_iov.
+ */
+#define dat_ep_post_fetch_and_add(ep, add_val, lbuf, cookie, rbuf, flgs) \
+ dat_extension( ep, \
+ DAT_EXT_FETCH_AND_ADD, \
+ (add_val), \
+ (lbuf), \
+ (cookie), \
+ (rbuf), \
+ (flgs))
+
+/*
+ * This asynchronous call is modeled after the InfiniBand atomic
+ * Compare and Swap operation. The cmp_value is compared to the 64 bit
+ * value stored at the remote memory location specified in remote_iov.
+ * If the two values are equal, the 64 bit swap_value is stored in
+ * the remote memory location. In all cases, the original 64 bit
+ * value stored in the remote memory location is copied to the local_iov.
+ */
+#define dat_ep_post_cmp_and_swap(ep, cmp_val, swap_val, lbuf, cookie, rbuf, flgs) \
+ dat_extension( ep, \
+ DAT_EXT_CMP_AND_SWAP, \
+ (cmp_val), \
+ (swap_val), \
+ (lbuf), \
+ (cookie), \
+ (rbuf), \
+ (flgs))
+
+#endif /* _DAT_EXTENSIONS_H_ */
+
Index: dat/common/dat_api.c
===================================================================
--- dat/common/dat_api.c (revision 4594)
+++ dat/common/dat_api.c (working copy)
@@ -1142,6 +1142,36 @@ DAT_RETURN dat_srq_set_lw(
low_watermark);
}
+DAT_RETURN dat_extension(
+ IN DAT_HANDLE handle,
+ IN DAT_EXT_OP ext_op,
+ IN ... )
+
+{
+ DAT_RETURN status;
+ va_list args;
+
+ if (handle == NULL)
+ {
+ return DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP);
+ }
+
+ /* verify provider extension support */
+ if (!dat_extensions)
+ {
+ return DAT_ERROR(DAT_NOT_IMPLEMENTED, 0);
+ }
+
+ va_start(args, ext_op);
+
+ status = DAT_EXTENSION(handle,
+ ext_op,
+ args);
+ va_end(args);
+
+ return status;
+}
+
/*
* Local variables:
* c-indent-level: 4
Index: dat/udat/udat.c
===================================================================
--- dat/udat/udat.c (revision 4594)
+++ dat/udat/udat.c (working copy)
@@ -66,6 +66,10 @@ udat_check_state ( void );
* *
*********************************************************************/
+/*
+ * Use a global to get an unresolved when run with pre-extension library
+ */
+int dat_extensions = 0;
/*
*
@@ -230,13 +234,44 @@ dat_ia_openv (
async_event_qlen,
async_event_handle,
ia_handle);
+
+ /*
+ * See if provider supports extensions
+ */
if (dat_status == DAT_SUCCESS)
{
- return_handle = dats_set_ia_handle (*ia_handle);
- if (return_handle >= 0)
- {
- *ia_handle = (DAT_IA_HANDLE)return_handle;
- }
+ DAT_PROVIDER_ATTR p_attr;
+ int i;
+
+ return_handle = dats_set_ia_handle (*ia_handle);
+ if (return_handle >= 0)
+ {
+ *ia_handle = (DAT_IA_HANDLE)return_handle;
+ }
+
+ if ( dat_ia_query( *ia_handle,
+ NULL,
+ 0,
+ NULL,
+ DAT_PROVIDER_FIELD_PROVIDER_SPECIFIC_ATTR,
+ &p_attr ) == DAT_SUCCESS )
+ {
+ for ( i = 0; i < p_attr.num_provider_specific_attr; i++ )
+ {
+ if ( (strcmp( p_attr.provider_specific_attr[i].name,
+ "DAT_EXTENSION_INTERFACE" ) == 0) &&
+ (strcmp( p_attr.provider_specific_attr[i].value,
+ "TRUE" ) == 0) )
+ {
+ dat_os_dbg_print(DAT_OS_DBG_TYPE_CONSUMER_API,
+ "DAT Registry: dat_ia_open () "
+ "DAPL Extension Interface supported!\n");
+
+ dat_extensions = 1;
+ break;
+ }
+ }
+ }
}
return dat_status;
Index: README
===================================================================
--- README (revision 4589)
+++ README (working copy)
@@ -1,5 +1,10 @@
There are now 3 uDAPL providers for openib (openib,openib_scm,openib_cma).
+NEW FEATURES for openib_cma provider:
+API extensions for immediate data and atomic operations have been added.
+see dat/include/dat/dat_extensions.h for new API's.
+see dapl/test/dtest/dtest_ext.c for example test cast
+
==========
1.0 BUILD:
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20051222/10a9f271/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: DAT_Extensions.pdf
Type: application/pdf
Size: 83940 bytes
Desc: not available
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20051222/10a9f271/attachment.pdf>
More information about the general
mailing list